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PREFACE 


"How  can  we  show  that  command  and  control  system  acquisitions 
will  contribute  to  increased  force  effectiveness?"  "The  most 
difficult  portion  of  the  Strategic  Defense  Initiative  is  battle 
management."  "How  do  we  organize  command  and  control  for  joint 
operations?"  "How  can  we  show  that  command  and  control  system 
acquisitions  will  contribute  to  increased  force  effectiveness?" 

These  are  the  types  of  problems  that  frequently  perplex  Department 
of  Defense  military  and  civilian  leaders,  analysts,  and  program 
managers.  Questions  of  this  type  require  the  establishment  of  an 
evaluation  structure  firmly  based  upon  Measures  of  Effectiveness 
(MOEs).  Such  measures  must  be  tailored  to  the  analytical  appli¬ 
cation  and  provide  credible  results  to  decisionmakers.  It  is  toward 
this  goal  that  a  Military  Operations  Research  Society  (MORS)  Work¬ 
shop  about  which  this  document  reports  was  conducted  at  the  Naval 
Postgraduate  School,  Monterey,  in  January  1985.  The  product  of  the 
Workshop  was  a  draft  of  this  report. 

The  reactions  to  the  draft  report  were  along  two  lines.  The 
first  was  that  the  product  was  too  abstract  and  remote  from  the 
practical  decisionmaking  process  to  be  utilitarian;  in  effect, 
interesting  theory,  but  so  what?  The  second  reaction,  especially 
among  the  participants,  was  now  that  we  have  agreed  on  a  structure 
for  understanding  Command  and  Control  (C^)  in  general,  can  we  apply 
it  to  solve  specific  problems?  These  reactions  amounted  to  viewing 
the  glass  of  methodological  utility  as  either  half  empty  or  half 
full.  Both  viewpoints  suggested  the  need  to  put  the  product  of  the 
first  Workshop  to  the  test.  That  was  done  in  January  1986  in  a 
second  MORS  Workshop,  "Evaluation  of  Command  and  Control  Systems," 

p 

which  examined  the  following  six  current  C  problems: 
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a. 

Army  Tactical 

b. 

Air  Force  Tactical 

c. 

Navy  Tactical 

d. 

Joint  Tactical 

e. 

Air  Force  Strategic 

f. 

Joint  Strategic 

The  problems  were  chosen  for  their  diversity  and  were  intended 
to  test  whether  the  theory  was  in  fact  widely  applicable  in  practice 
The  attendees  at  the  1986  Workshop  were  a  mixture  of  those  who 
attended  in  1985  and  first-time  participants  who  were  selected  on 

the  basis  of  their  knowledge  of  the  particulars  of  one  of  the  six 
problems. 

Was  the  consensus  an  overwhelming  endorsement  of  this  report 
and  its  recommended  structure  for  evaluating  C  systems?  That  would 
be  too  strong  a  claim.  At  one  extreme  was  an  unqualified  approval 
and  a  conviction  that  as  experience  was  gained  and  the  evaluation 
technique  applied,  it  would  prove  to  be  strong  and  reliable.  At  the 
other  extreme  was  a  tentative  endorsement  that  the  theory  was  better 
than  nothing  (that  it  could  be  worse  than  nothing  was  viewed  as  a 
possibility  at  the  outset)  and  also,  it  was  at  least  useful  in 
focusing  the  analysis  and  structuring  and  evaluation.  No  one  felt 
that  the  application  of  the  methods,  measures  of  effectiveness,  or 
models  was  counterproductive,  too  rigid  or  formalized,  or  inherently 
defective  in  the  light  of  the  test  on  their  problem  from  among  the 
six.  The  general  flavor  of  the  criticisms  was  either  that  the 
guidance  provided  in  this  report  would  need  to  be  adapted,  modified, 
or  relaxed  to  fit  each  particular  case,  or  that  the  guidance  needed 
further  development,  amplification,  and  detail.  Since  these  two 
criticisms  are  antithetical,  perhaps  the  report  has  struck  about  the 
right  balance. 
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The  1986  report  now  in  your  hands  is  an  improved  version  of  the 
1985  draft.  It  incorporates  a  small  number  of  changes  proposed 
during  internal  and  external  review,  as  well  as  the  correction  of 
technical  errors.  In  a  few  places  it  benefits  from  the  results  of 
the  1986  Workshop. 

The  MORS  and  Workshop  participants  have  made  this  report 
available  in  the  expectation  that  it  will  be  a  useful  guide.  The 
content  as  regards  definitions,  selection  and  application  of  MOEs, 
the  structure  of  models,  and  the  formulation  of  mathematical 
analysis  should  serve  as  a  common  point  of  departure  and  aid  in 
establishing  a  common  language  and  set  of  analytical  tools  for 
analysts,  whether  their  interest  is  research,  development, 
procurement,  or  operation  of  a  system.  In  particular,  the  report 
contains  the  Modular  Command  and  Control  Evaluation  Structure 
(MCES).  The  MCES  is  designed  to  guide  C^  systems  evaluation  and 
architecture  development  within  the  Department  of  Defense. 
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CHAPTER  1 


INTRODUCTION 

by 

Edward  C.  Brady 
Dr.  Morton  L.  Metersky 
Dr.  Ricki  Sweet 


1.0  FOREWORD 

p 

Command  and  Control  (C  )  is  central  to  warfighting.  A  great 
investment  is  being  made  to  improve  the  application  of  automation 

p 

and  communications  technology  to  C  .  The  incoming  Reagan 
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administration's  number  1  defense  priority  was  C  .  But  what  does  C 
do?  How  do  we  know  it  does  what  we  think  or  hope  it  does?  Does 
more  of  it,  or  improvements  in  it,  pay  off  by  increasing  force 
effectiveness?  These  questions  place  the  issue  directly  into  the 
realm  of  evaluation. 

C^  is  an  interdisciplinary  field.  There  is  no  single  pro- 

2 

fessional  community  that  encompasses  both  aspects  of  the  C 
environment,  the  operational  and  the  technological.  In  the 
operational  environment,  people  are  the  most  important  element,  but 
technological  advances  have  diverted  attention  to  hardware. 

The  past  has  been  characterized  by  a  lack  of  agreement  among 
analysts  and  decisionmakers  over  a  number  of  important  issues. 

Among  these  were: 


a.  The  use  of  a  set  of  consistent  definitions  relevant  to  both 
C”^  systems  and  the  measurement  thereof. 

p 

b.  The  relationship  between  a  C  process  and  the  physical 
entities  that  are  part  of  the  system. 

2 

c.  The  specification  of  an  appropriate  model  of  C  . 

2 

d.  The  appropriate  integration  of  the  selected  C  model, 
measures,  methods,  and  mathematics. 
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e.  The  way  in  which  the  model  may  be  incorporated  in  a 
specific  problem. 

f.  The  relationship  between  the  decision  to  be  supported  and 
the  analysis  itself. 

It  appears  to  us  that  what  is  needed  is  an  integrated,  balanced 

2 

approach.  This  report  initiates  the  establishment  of  such  a  C 

2 

evaluation  approach.  Such  a  roadmap  for  the  evaluation  of  C 
systems  should  be  useful  in  the  determination  of  both  the 
contribution  of  systems  to  warfighting  capability  and  the 
selection  of  a  candidate  system  or  architectural  configuration 
from  among  competing  alternative  designs. 

This  report  is  written  in  the  hope  that  it  will  provide  a  ready 
reference  to  the  understanding  and  use  of  evaluation  measures  for 
C2. 

Readers  may  be  decisionmakers  involved  in  budgetary  or 
programmatic  decisions,  design  engineers,  or  operational  officers. 

It  is  believed  that  a  Measures  of  Effectiveness  (MOEs)  concept  may 
also  help  non-analysts  understand  what  such  studies  mean  to  them. 
Unfortunately,  the  analytic  community  has  not  always  articulated  its 
technical  framework  and  results  in  a  manner  which  can  be  safely  and 
unambiguously  interpreted  by  the  layman.  Careful  definition  and 
structuring  of  analyses  lead  to  results  which,  when  expressed 
clearly,  can  be  readily  understood. 

To  that  end,  it  is  our  intention  that  this  report  will  provide 
readers  with: 

a.  A  compendium  of  terminology  and  references. 

2  2 

b.  An  important  partition  of  the  field  into  C  process  and  C 
system. 

c.  A  structure  of  the  field  of  MOEs  (dimensions). 

d.  A  conceptual  model(s)  of  the  process  and  the  function  of 
MOEs  within  the  model. 
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e.  An  appreciation  of  the  complexity  of  mathematically 
modeling  and  its  MOEs. 

f.  An  understanding  of  the  data  collection  necessary  to 
specify  MOEs, 

g.  Exposure  to  the  application  of  MOEs. 

1 .1  Background 

Important  efforts  have  been  made  by  small  groups  of  specialists 
over  the  last  decade  in  broadening  our  perspective  on  both  the 
substance  of  C  and  the  evaluation  of  systems  created  to  carry  out 
this  function.  As  a  result,  a  meaningful  beginning  has  been  made  in 
the  development  of  a  theory  of  and  its  relationship  to  a  theory 
of  combat.  These  efforts  have  been  scattered  among  all  parts  of  the 
military  establishment.  While  there  have  been  periodic  attempts  to 
integrate  the  results  of  these  activities  into  some  comprehensive 
form,  it  has  usually  not  been  possible  to  document  and  disseminate 
these  efforts.  Thus  the  community  dealing  with  these  issues  has 
continued  to  express  a  need  for  a  reasonably  accessible  source  of 
information  conveying  state-of-the-art  knowledge  of  such  matters. 

The  initial  impetus  for  this  effort  was  one  of  those 
integrating  attempts  referred  to  above.  It  was  triggered  by  a 
challenge  from  General  Eaglet  in  his  role  as  Deputy  Chief  of  Staff, 
Plans  and  Programs,  Headquarters,  Air  Force  Systems  Command. 
Specifically,  General  Eaglet  invited  Air  Force  planners  to  determine 
the  force  effectiveness  of  systems.  This  meeting,  called  the 
"Measures  of  Effectiveness  for  Evaluation  Symposium,"  took  place 
at  The  MITRE  Corporation  in  Bedford,  Massachusetts,  on  February  28 
to  March  1 ,  1 984. 

It  seemed  appropriate  to  the  symposium's  organizers  to  direct 
the  expert  knowledge  of  the  analytic  community  to  arrive  at  a 
synthesis  of  the  current  state  of  the  art.  The  symposium's  general 
goals  were  established  to: 
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a. 


Determine  a  baseline  of  common  principles,  Including; 


1  .  Definitions. 

2.  Approaches. 

3.  Conceptual  models. 

b.  Identify  what  else  is  known. 

c.  Determine  what  needs  to  be  learned. 

In  a  series  of  organizational  meetings,  the  1984  Symposium 
chairpersons.  Dr.  Ricki  Sweet  and  LTC  Thomas  Fagan  III,  and  working 
group  session  chairpersons.  Dr.  Zitta  Z.  Friedlander,  The  MITRE 
Corporation,  Griffin  F.  Hamilton,  EASTAN  Corporation,  Linda  Hill, 
SAIC,  Dennis  Holstein,  LOGICON,  and  Richard  Hu,  Naval  Sea  Systems 
Command,  developed  the  method  used  to  address  the  overall 
objective.  Four  working  groups  were  formed,  each  to  discuss  in 
parallel  the  same  topics.  The  topics  were: 

a.  Working  Definitions. 

b.  MOE  Identification. 

c.  Evaluation  Techniques. 

Panels  were  formed  to  address  the  convened  Symposium  after  each 
working  session.  Each  working-group  chairperson  presented  the 
results  of  their  deliberations.  On  the  first  day,  it  became  evident 
that  simple  solutions  would  not  emerge  from  the  deliberations  of 
such  complex  topics.  Therefore,  a  fourth  topic.  Summary,  was  added 
to  include  an  overall  appraisal  of  the  current  status  and  future 
course  of  this  type  of  MOE  analysis. 

Deliberations  of  the  1984  Symposium  were  reported  to  the  52nd 
Military  Operations  Research  Society  (MORS)  Measures  of 
Effectiveness  Working  Group  in  June  1984,  with  an  audience  of  over 
100  attendees.  Presentations  were  made  by  the  working-group 
chairperson  as  well  as  LtCol  Edward  C.  Jonson,  Director  of  Long 
Range  Planning,  ESD/XR,  Ted  Jarvis,  The  MITRE  Corporation,  and 
Dr.  Morton  L.  Meter sky.  Naval  Air  Development  Center. 


Based  upon  the  critiques  and  discussions  in  the  Measures  of 
Effectiveness  Working  Group,  it  was  suggested  MORS  sponsor  an 
interim  workshop  for  selected  members  and  the  analytic  community. 

The  "organizating  committee,"  Dr.  Ricki  Sweet,  also  the  chairperson, 
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MORS  C-^  Working  Group,  Dr.  Michael  G.  Sovereign,  Chairman,  Joint  C-^ 

Program,  Naval  Postgraduate  School  (NPS),  and  Dr.  Morton  L.  Metersky, 

2 

developed  a  proposal  for  a  MORS-sponsored  C  MOE  Workshop. 

This  Workshop  was  chaired  by  Dr.  Sweet,  co-chaired  by 
Dr.  Metersky,  and  hosted  by  Dr.  Sovereign.  It  was  convened  at  the 
Naval  Postgraduate  School  in  January  1985.  In  addition  to  the 
organizing  committee,  several  other  key  people  were  heavily  involved 
in  the  Workshop  and  in  the  development  of  this  document.  They  were 
Dr.  William  Foster,  The  MITRE  Corporation,  Dr.  Stuart  Brodsky, 

Sperry  Corporation,  Walker  Land,  IBM,  Richard  Miller,  now  OSD, 

Charles  Smith,  now  Nichols  Research  Corporation,  and  Dr.  Conrad 
Strack,  Defense  Systems,  Incorporated. 

These  people  also  participated  in  the  development  of  a 

"strawman"  which  guided  the  deliberations  of  the  MORS-NPS 

Workshop.  Using  this  strawman  under  the  direction  of  this  group, 

2 

the  Workshop  attempted  to  engage  both  analysts  and  C  theorists  in  a 

structured  dialogue  on  the  development,  computation,  and  application 
2 

of  C  MOEs  as  tools  for  the  evaluation  process. 

During  the  Workshop  a  cadre  of  designated  Intergroup 
Coordinators  (ICs)  provided  the  interface  for  the  working  groups, 
each  of  which  worked  on  an  independent  portion  of  the  problem.  The 
major  function  of  the  ICs  was  to  bring  ideas,  thoughts,  and  concepts 
to  the  attention  of  other  working  groups  by  circulating  among  them 
as  they  saw  fit.  Table  1-1  provides  a  list  of  Workshop 
participants.  ICs  are  indicated  therein.  Each  participant  also 
contributed  materially  to  the  preparation  of  this  document  during 
the  Workshop. 
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TABLE  1-1 


WORKSHOP  PARTICIPANTS 


Dr.  Ricki  Sweet,  MITRE  Chair 

Dr.  Morton  L.  Metersky,  NADC  Co-Chair 

Dr.  Michael  G.  Sovereign,  NPS  Host 

Applications  Working  Group 

Mathematics  Working  Group 

Dr.  William  Foster,  MITRE,  Chair 
COL  Robert  Allison,  USA,  JCS 
Robert  Choisser,  DCA 

MAJ  Bernard  Galing,  USA,  TREM 
LtCol  Edward  C.  Jonson,  USAF,  ESD 
Dr.  S.  Z.  Mikhail,  NOSC 

MAJ  Larry  Rhoads,  USMC,  MCTSSA 

Dr.  Stuart  Brodsky,  Sperry,  Chair 

Dr.  Alex  Levis,  MIT 

Dr.  Tony  Richardson,  Daniel  Wagner 
Associates 

Dr.  Conrad  Strack,  DSI 

Dr.  Edison  Tse,  Stanford 

Dr.  Clairice  Veit,  RAND 

Conceptual  Model  Working  Group 

Intergroup  Coordinators 

Walker  Land,  IBM,  Chair 

Ted  Bean,  MITRE 

Leon  Godfrey,  CAORA 

Judy  Grange,  SAIC 

LCDR  Don  Newman,  NAVAIR 

Tony  Snyder,  RADC 

Measures  Working  Group 

Edward  C.  Brady,  MITRE 

Dr.  Norval  Broome,  MITRE 

Dr.  John  Dockery,  JCS 

Dr.  Zitta  Friedlander,  MITRE 

CDR  Paul  Girard,  ONR 

Griffin  Hamilton,  EASTAN 

Dr.  Joel  S.  Lawson,  Jr.,  NAVELEX 

Dr.  Martin  Leonardo,  NADC 

Richard  Miller,  TORA,  Chair 

Dr.  Harold  Glazer,  MITRE 

Linda  Hill,  SAIC 

Charles  Smith,  ANSER 

Capt  Bruce  Thieraan,  USAF,  JCS 

*A11  affiliations  as  of  meeting  date. 
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A  staff  provided  the  necessary  clerical  and  administrative 
support.  This  staff  included  Natalie  Addison,  MORS,  Barbara  Walker, 
Naval  Air  Development  Center,  Zanie  Bactad,  NFS,  Major  Bernard 
Galing,  NFS,  and  Commander  Joseph  Stuart,  NFS. 

The  essence  of  these  deliberations  was  briefed  at  the  53rd  MORS 
in  June  1985.  Viewgraphs  used  at  the  briefing  can  be  found  in 
Appendix  A.  Review  of  the  preliminary  materials  was  solicited  from 
the  attendees,  members  of  the  MORS  Board  of  Directors,  participants 
of  the  Workshop,  and  other  interested  parties. 

This  report  represents  the  current  thinking  of  the  experts  who 
participated  on  a  voluntary  basis  in  the  two  meetings  described.  As 
an  integrative  presentation,  it  should  be  viewed  as  an  initial 
formulation  of  the  necessary  components  of  an  evaluation 

p 

architecture  for  C  .  When  expanded  upon,  this  report  and  successor 
documentation  will  represent  the  explication  and  application  of  the 

p 

concept  and  methodology  for  the  evaluation  of  C  systems. 

1 . 2  Objectives 
1.2.1  This  Report 

2 

Evaluations  of  C  systems  are  undertaken  for  a  variety  of 
different  purposes.  Therefore,  it  is  expected  that  this  report  will 
be  of  interest  to  several  different  groups  of  readers,  categorized 
as  follows: 


a.  Analysts  working  in  the  Command,  Control,  and  Commu¬ 
nications  (C^)  field.  Experienced  analysts  are 
already  aware  of  the  problems  that  are  raised  but  may 
benefit  from  their  compilation  as  a  reference.  Beginning 
analysts  could  consider  it  as  part  of  their  education. 


b.  Those  who  work  in  design,  development,  and  test  functions 
producing  and  using  C^  evaluation  measures  and  who  then 
need  to  be  familiar  with  their  context. 
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c.  Decisionmakers  in  the  field,  involved  in  doctrine, 

development  and  force  structure,  and  operational  planning 
and  execution  who  need  to  be  aware  of  the  options  available 
for  measurement,  the  general  nature  of  the  difficulties 
and,  most  importantly,  the  caveats  concerning  the  use  of 
such  measures  that  should  be  brought  to  their  attention. 

1.2.2  Overall  Effort 

There  are  many  uses  for  MOE  analyses.  The  tendency  of  military 
system  development  agencies  is  to  use  MOEs  while  seeking  answers  to 
acquisition  issues.  Research  and  Development  (R&D)  organizations 
look  to  MOEs  as  a  way  to  validate  systems  design  and  to  correct 
current  and  long-term  deficiencies.  User  commands  need  solutions  to 
operational  problems  and  look  to  MOEs  as  a  way  to  help  them  in  this. 

Programmatic  decisions  starting  in  the  conceptual  development 
phase  determine  whether  or  not  a  system  will  be  funded.  MOEs 
play  a  part  in  providing  the  necessary  analysis  supporting  decisions 
to  approve  funding  for  programs.  They  provide  military  system 
development  agencies  with  a  tool  to  help  obtain  approval  for 
programs,  and  provide  a  link  to  the  assurance  of  whether  a  system 
will  fulfill  operational  requirements. 

Acquisition  management  activities  have  a  continual  concern  for 
the  proper  use  and  interpretation  of  performance  measures.  This  is 
true  at  the  component  equipment  level  and  in  the  evaluation  of  the 
contributions  of  such  components  to  the  performance  of  large 
aggregates  of  equipment,  the  interaction  of  said  aggregates  and 
people,  and  the  contribution  of  these  to  an  ability  to  carry  out  a 
mission. 

Design  decisions  can  also  be  significantly  affected  by  data 
derived  from  measurements.  These  data  can  provide  an  indication  of 
how  well  a  proposed  system  can  perform  and  what  design  changes  may 
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be  desirable  to  effect  increases  in  capability.  Whether  this 
information  is  derived  from  laboratory  test  and  evaluation, 
operational  test  and  evaluation,  or  from  feedback  of  exercises  or 
operations,  it  frequently  involves  the  use  of  MOEs  and  supports 

p 

efforts  to  improve  C  systems. 

p 

Finally,  MOEs  can  be  applied  to  procedural  or  operational  C 
systems.  Through  evaluation  data  and  feedback,  operational 
commanders  and  developers  of  tactical  and  strategic  doctrine  can 
obtain  an  indication  of  the  effectiveness  of  existing  doctrine  and 
can  propose  modifications  to  improve  capability.  Design 
recommendations  to  development  agencies  can  be  another  result  of  MOE 
analyses.  Judicious  use  of  operational  systems  requires  an 
understanding  of  how  to  measure  performance  and  relative 
contributions  to  mission  success.  It  also  requires  an  appreciation 
of  how  such  measures  may  vary  depending  upon  alternative 
arrangements  and  uses  of  systems  and  upon  different  operational 
contexts.  This  type  of  knowledge  coupled  with  attention  to 
deficiencies  provides  a  basis  for  doctrine  development  and  force 
structure  activities. 

It  is  not  surprising  that  the  need  for  a  global  model  of  and 
associated  MOEs  is  not  completely  accepted.  Decisionmakers  may  look 
to  MOEs  to  reinforce  their  particular  needs.  This  frequently 
results  in  an  emphasis  on  a  single-purpose  model  in  contrast  to  a 
general  conceptual  model.  Program  or  acquisition  managers,  for 
example,  deal  with  present  systems  or  those  in  development  in  a 
specific  mission  area.  Therefore,  they  are  most  concerned  with 
evaluation  of  systems  under  procurement  or  in  operation.  They  are 
interested  in  improving  effectiveness  of  the  system  being  procured, 
or  in  the  case  of  an  operational  system,  assuring  that  the  system 
will,  for  example,  increase  its  ability  to  leverage  offensive  and 
defensive  weapons  systems.  These  immediate  concerns  will  always  be 
in  opposition  to  the  more  universal  concern  for  the  global  models 
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and  their  associated  MOEs.*  Overall,  MOE  analysis  must  deal  with 
long-range  global  problems,  as  well  as  near-term  issues  for  design, 
programmatic,  and  operational  applications.  Table  1-2  summarizes 
the  effects  of  MOEs  on  decisions. 

While  evaluative  measures  are  continuously  used  In  the  above 
contexts,  as  well  as  in  several  other  related  areas,  it  is  normally 
the  case  that  an  individual  analyst  or  decisionmaker  only  needs  to 
use  a  subset  of  such  measures  for  any  particular  problem  at  hand. 
This  frequently  results  in  an  emphasis  on  a  single  interpretation  of 
effectiveness  measures  in  contrast  to  a  broadly  based  perspective  on 
such  measures.  In  fact,  many  users  of  such  measures  do  not 
recognize  the  need  for  a  broadly  based  understanding  of  a  theory  of 
such  measures.  Without  the  uniform  broad  theory  to  draw  upon,  and 
with  the  practical  need  to  get  the  problems  being  addressed  solved, 
most  analyses  must  take  the  pragmatic  approach  of  a  narrowly 
constrained  bottoms-up  perspective.  Our  objective,  to  provide  a 
cohesive  top-down  perspective,  should  lead  to  better  evaluation  for 
less  time  and  effort. 

1 .3  Overview  of  This  Document 

This  document  is  organized  in  a  manner  similar  to  the  format 
used  in  the  Workshop.  Thus  it  is  a  collection  of  chapters  produced 
by  committees.  For  this  reason,  references  to  existing  material  are 
not  provided  despite  their  relevance.  Coordination  of  the  chapters 
was  specifically  attempted  but  was  not  completely  effective. 
Differences  in  nuance  and  perspective  are  still  evident.  However, 
there  is  an  overall  structure.  Following  the  introduction  in 
Chapter  1 ,  Chapter  2  presents  definitions,  and  the  overall  archi¬ 
tecture  is  discussed  in  Chapter  3.  Chapter  4  directs  attention  to 
the  decisionmaker's  need  for  C^  measures  in  his  applications 
environment,  while  Chapter  5  presents  the  conceptual  process  model 


*MOE  is  used  here  in  a  generic  sense. 
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TABLE  1-2 


EFFECT  OF  MOEs  ON  DECISIONS 


ANALYSIS  OBJECTIVES 

EFFECTS 

USED  BY 

Concep tual 
Development 

Planning  and  Doctrine 
System  Requirements 

Systems,  Operational 
Commands,  Requirements 
Analysts 

Acquisition 

Procurements 

System  Specifications 

System  Commands 

Program  Managers 

Design 

System  Capability 

Choice  of  Technology 
Redundancy 

R&D  Scientists 

Design  Engineers 

System  Commands 

Operations 

Development  of 

Doctrine  and  Tactics 

Operational  Commands 
and  Commanders 

Measuring  Warfighting 
Capability 

System  Deficiencies 

Operational  Commanders 
Operational  Planners 
Schools,  System  Sponsors 
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of  C^.  This  is  the  basis  for  the  work  that  follows.  It  establishes 
the  functions  that  must  be  performed  and  evaluated.  Chapter  6 
frames  the  model  with  the  analytical  or  evaluation  process, 
discusses  techniques  for  obtaining  measurements,  and  gives 
examples.  Chapter  7  provides  a  theoretical  statement  of  the  problem 
which  gives  guidance  on  the  approach  to  measurement  regarding 
sufficiency  and  precision.  A  short  summary  is  presented  in  Chapter  8. 
Finally,  Appendix  A  contains  briefing  materials  used  at  the  53rd 
MORS.  Appendix  B  presents  a  list  of  the  acronyms  which  we  have 
attempted  to  use  minimally  in  this  document. 
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CHAPTER  2 


DEFINITIONS 

by 

Edward  C.  Brady 
Dr.  Morton  L.  Meter sky 
Dr.  Michael  G.  Sovereign 


2.0  INTRODUCTION 

Within  the  military  analytic  community  there  have  been  long¬ 
term  debates  regarding  an  appropriate  definition  for  the  terms 
"measures  of  effectiveness"  and  "command  and  control."  It  is 
unlikely  that  the  efforts  of  this  Workshop  resolved  these  issues. 
Nonetheless,  there  was  need  among  the  participants  and  in 
communicating  the  efforts  of  the  Workshop  to  others  to  have  an 
understanding  of  what  these  terms  mean.  Therefore,  definitions  are 
presented  for  the  following  terms:  command  and  control,  C^  systems, 
physical  entities,  structure,  C  process,  dimensional  parameters, 
measures  of  performance,  measures  of  effectiveness,  and  measures  of 
force  effectiveness.  These  definitions  were  accepted  and  used  by 
most  of  the  Workshop  participants. 

2. 1  Command  and  Control 

The  military  activity  of  interest  to  the  Workshop  was  "command 
and  control."  Command  and  control  has  been  used  as  a  broad  concept 
whose  breadth  has  been  denoted  by  the  use  of  terms  such  as  C^;  C^ ; 
Command,  Control,  Communications,  and  Computers;  and  Command, 
Control,  Communications,  and  Intelligence  (C^I).  Over  time  and  in 
different  parts  of  the  military  community,  this  term  has  also  had  a 
variety  of  narrower  definitions  (for  example,  sometimes  the  distinc¬ 
tion  is  made  between  "command  and  control"  and  "combat  direction," 
and  sometimes  a  distinction  is  made  on  the  basis  of  what  is 
"embedded"  or  "non-embedded"  in  sensor  and  weapons  platforms  or 
systems).  It  is  the  intent  of  this  Workshop  to  consider  "command 
and  control"  in  as  broad  a  generic  sense  as  possible  in  the  belief 
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that  a  broadly  based,  generic,  top-down  approach  will  be  of  value  to 

the  analytic  community  and  to  the  readers  and  users  of  analytical 

2 

reports.  Thus,  throughout  this  report  the  term  "C  "  should  be  taken 

to  mean  as  broad  a  concept  as  is  useful  for  the  analysis  being 

undertaken  or  discussed.  This  means  that  there  will  be  some 

variation  of  its  definitional  boundaries  depending  upon  the 

analytical  questions  being  pursued.  This  is  common  practice  in 

almost  all  forms  of  military  analysis.  It  is  agreed  that  this  term 

does  not  include  weapons  functions,  but  it  may  or  may  not  include 

sensor  functions.  It  is  not  felt  that  this  flexibility  obstructs 

?  2 

progress  in  developing  a  theory  of  C  ,  a  model  of  C  ,  or  the 
development  and  use  of  analytical  measures.  (Several  Workshop 
participants  did  not  accept  this  view). 

The  spirit  of  this  definition  is  well  captured  by  the  approved 
Department  of  Defense  (DOD)  definition  found  in  Joint  Chiefs  of 
Staff  (JCS)  Pub  1: 

"The  exercise  of  authority  and  direction  by  a  properly 
designated  commander  over  assigned  forces  in  the 
accomplishment  of  his  mission.  Command  and  control 
functions  are  performed  through  an  arrangement  of 
personnel,  equipment,  communications,  facilities,  and 
procedures  which  are  employed  by  a  commander  in  planning, 
directing,  coordinating  and  controlling  forces  and 
operations  in  the  accomplishment  of  his  mission." 

2.2  C^  Systems 

2 

Furthermore,  it  is  felt  that  in  order  to  theorize  about  C  and 

2  2 
to  construct  a  model  of  C  ,  it  is  useful  to  have  a  definition  of  "C 

systems."  It  is  thought  to  be  important  for  evaluation  applications 

that  this  concept  be  viewed  as  having  three  components:  "physical 

2 

entities,"  "structure,"  and  "C  process." 
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a*  Physical  Entitles  -  Refers  to  equipment  (computers  and 

peripherals,  modems,  jammers,  antennas,  computer  local-area 
networks),  software,  facilities,  and  people. 

b.  Structure  -  Identifies  the  arrangement  and 
interrelationships  of  physical  entities,  procedures, 
protocols,  concepts  of  operation,  and  information 
patterns.  (This  frequently  reflects  doctrine  and  may  be 
scenario  dependent.)  Such  arrangements  are  often  spatial 
and  temporal . 

The  JCS  Pub  2  definition  of  a  command,  control,  and 
information  system  captures  the  intent  of  our  definition  of 
systems: 

"An  integrated  system  comprised  of  doctrine, 
procedures,  organizational  structure,  personnel, 
equipment,  facilities,  and  communications  which 
provides  authorities  at  all  levels  with  timely 
and  adequate  data  to  plan,  direct  and  control 
their  operations." 

2 

c.  C  Process  -  In  addition,  it  is  important  to  understand 
that  "C^  process"  is  "what  the  system  is  doing"  and 
reflects  functions  carried  out  by  the  system— sensing, 
assessing,  generating,  selecting  alternatives,  planning  and 
directing. 

2.3  Boundaries 

The  "boundary  of  a  system"  is  defined  as  a  function  of  the 
analysis  at  hand,  and  is  the  delineation  between  the  system  being 
studied  and  the  environment.  Thus,  while  the  definitions  are 
reasonably  rigorous,  they  apply  in  the  context  of  the  system 
boundary  (including  its  environment).  An  MOP  in  one  analysis  might 
well  become  an  MOE  in  another.  For  example,  if  the  National  Command 
Authority  and  its  information  system  (the  Worldwide  Military  Command 
and  Control  System)  is  the  C  system  being  evaluated  and  a  conven¬ 
tional  war  is  being  fought,  the  force  elements  of  the  battle,  e.g., 
battle  groups  at  sea,  corps  in  the  field,  and  wings  of  aircraft,  can 
be  viewed  as  weapons  systems.  Thus,  MOFEs  would  be  relative  to  the 
armed  forces  achieving  their  mission  in  some  theater  of  action.  In 
another  analysis,  each  of  these  "weapons  systems"  alone  could  become 
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the  force  which  is  accomplishing  its  unique  mission  and  to  which  an 
analyst  must  relate  his  MOEs.  The  Figure  2-1  diagram  depicts  the 
relationship  between  MOEs  and  MOFEs. 

2.H  Measures 

The  analytical  activity  of  interest  to  the  Workshop  was 
measuring  and  evaluating  the  behavior/performance  of  the  C  system 
in  a  context  appropriate  to  the  problem  being  evaluated.  To  be 
generic,  it  is  essential  that  postulated  measurements  be  adapted  to 
the  analytical  question  being  pursued  and  to  the  boundary 
definitions  of  the  system  being  investigated.  In  dealing  with 
this  issue  the  analytic  community  has  developed  a  set  of  terms 
related  to  one  another: 

a.  Dimensional  Parameters. 

b.  Measures  (Variables)  of  Performance  (MOPs). 

c.  Measures  of  (C^  System)  Effectiveness  (MOEs). 

d.  Measures  of  Force  Effectiveness  (MOFEs). 

The  three  components  describing  a  system  must  operate  within 
an  operational  and  physical  environment  which  is  represented  in 
Figure  2-1  as  outside  the  force  boundary.  Effects  of  environmental 
considerations  are  subsumed  within  these  components. 

Consensus  has  not  been  achieved  on  how  these  measures  can  be 
stringently  defined  so  as  to  be  comprehensive  and  distinguishable 
from  one  another.  Therefore,  the  following  definitions  were 
presented  for  use  in  the  Workshop: 

2 

a.  Measured/Specified  Inside  the  Boundary  of  the  C  System: 

1 .  Dimensional  Parameters  -  Properties  or  characteristics 
inherent  in  the  physical  entities  whose  values 
determine  system  behavior  and  the  structure  under 
question  even  when  at  rest  (size,  weight,  aperture 
size,  capacity,  number  of  pixels,  luminosity). 


ENVIRONMENT 


WHERE: 

D  =  DIMENSION 
P  =  PERFORMANCE 
E  =  EFFECTIVENESS 
F  =  FORCE  OUTCOME 


NOTE:  TIME  CAN  BE  A  PARAMETER  OF  ALL  (D,  P,  E,  &  F) 


FIGURE  2-1 

MEASURE  RELATIONSHIPS 
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2.  MOP  “  Closely  related  to  inherent  parameters  (physical 
and  structural)  but  measure  attributes  of  system 
behavior  (gain  throughput,  error  rate,  signal- to-noise 
ratio) . 

b.  Measured/Specified  Outside  the  Boundary  of  the  System; 

1 .  MOE  -  Measure  of  how  the  system  performs  its 
functions  within  an  operational  environment 
(probability  of  detection,  reaction  time,  nmber  of 
targets  nominated,  susceptibility  of  deception). 

c.  Measured/Specified  Outside  the  Boundary  of  the  For_ce: 

1 .  MOFE  -  Measure  of  how  a  system  and  the  force 

Tsensors,  weapons,  system,  and  structure)  of  which 
it  is  a  part  performs  missions  (contribution  to  battle 

outcome) . 

MOEs  are  measured  relative  to  some  standard,  which  is  often 
implicit.  Often  implied  is  "how  a  perfect  system  would  perform," 
i.e,,  the  probability  of  detection  compared  to  the  number  of 
detections  theoretically  possible,  giving  a  perfect  system  a 
probability  of  1.  Varieties  of  such  standards  are  used,  including, 
for  example,  how  a  "baseline"  system  performs,  or  compared  to 
mission  requirements. 

A  distinction  is  made  between  the  terms  MOE  and  MOFE.  An 
implication  of  this  distinction  is  that  many  other  factors 
contribute  to  whether  an  improvement  in  system  MOEs  results  in 
improvements  in  MOFEs.  For  example,  increasing  target  detections 
when  no  further  ammunition  is  available  to  weapons,  or  increasing 
the  rate  of  target  detections  when  the  rate  of  fire  cannot  be 
increased,  or  improving  post-strike  connectivity  to  weapons  which 
are  all  vulnerable  to  first-strike  destruction  will  not  improve 

MOFEs. 
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Relating  MOEs  to  MOFEs  and  thereby  evaluating  systems  is  a 
very  complex  issue.  As  one  issue  to  be  accounted  for,  it  is  noted 
that  MOEs  themselves,  as  well  as  MOFEs,  are  related  to  the 
operational  context  and  to  assumed  enemy  actions.  As  such,  they  are 
inherently  scenario  dependent. 

This  also  means  that  it  may  be  the  case  that  measures  used  for 

one  purpose,  e.g.,  acquisition  management,  are  inappropriate  to 
2 

evaluate  C  system  performance  for  another  purpose,  e.g.,  doctrine 
development.  It  is  unlikely  that  a  single  set  of  measures  can  be 
used  in  each  application,  or  that  each  measure  can  be  used  in  every 
application.  Therefore,  the  measures  must  be  carefully  chosen  from 
among  potential  candidates  and  related  to  supporting  decisions  to  be 
made. 

2.5  Summary 

In  most  cases,  parameters,  when  related  to  physical  entities, 
are  as  objective  and  quantified  as  they  would  be  in  a  hard  science 
or  engineering  sense,  and  can  be  measured  or  estimated.  MOPs  also 
sometimes  are  subjective  and  qualitative,  e.g.,  ordinal  ranking  by 
"experts,"  and  may  or  may  not  be  assigned  numerical  values.  MOEs 
and  MOFEs  are  heavily  judgmental  even  vSien  tiiey  are  numerical,  since 
choosing  system  boundaries,  particular  functions  to  be  evaluated, 
and  the  reference  standards,  and  making  other  such  judgments  can 
greatly  influence  particular  numerical  calculations.  Even  when 
based  on  models,  they  are  highly  dependent  on  the  model  assumptions, 
simplifications,  values  of  input  parameters,  and  the  selection  of 
output  measures  to  be  estimated. 

P  P 

In  summary,  working  definitions  have  been  suggested  for  C  ,  C 

2 

systems,  physical  entities,  structure,  C  process,  and  tiieir  related 
measures  (dimensional  parameters,  measures  of  performance,  measures 
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of  effectiveness,  and  measures  of  force  effectiveness).  Addition¬ 
ally,  it  has  been  emphasized  that  these  measures  are  distinguished 
by  where  they  are  "measured"  relative  to  the  boundary  of  the  system 
under  consideration,  including  evaluation  scenario  and  other 
environmental  factors  such  as  hostile  capabilities,  assumptions 
about  intelligence  and  knowledge,  etc.,  which  are  also  discussed. 
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CHAPTER  3 


EVALUATIVE  STRUCTURE  -  AN  ARCHITECTURE 
by 

Dr.  Ricki  Sweet 


3.0  INTRODUCTION 

The  evaluation  "Architecture,"  as  used  in  this  chapter,  refers 
to  the  relational  structure  between  building  blocks  established  for 
this  Workshop  and  discussed  in  the  remaining  chapters  in  this 
report:  "Applications  and  the  Need  for  C^  Measures,"  "Model," 
"Measures,"  and  "Mathematics."  The  building  blocks  themselves  are 
generic.  In  any  specific  evaluation,  the  generic  block  will  be 
replaced  by  alternatives  which  meet  stated  criteria  for  each  block 
in  the  analysis. 

This  chapter  provides  an  extensive  summary  of  the  chapters 
following.  A  preliminary  version  of  this  chapter  appears  in  the 
June  1985  Phalanx.  The  four  working  groups  at  the  Workshop  each 
addressed  a  different  part  of  this  "Architecture"  for  the  evaluation 

p 

of  C  systems. 

The  boundaries  for  any  study  are  set  by  the  "Applications." 

The  general  conceptual  model  specifies  the  range  of  contexts  within 
which  problems  may  be  analyzed;  thereupon  the  process  model 
addresses  the  functionality  of  interest.  A  set  of  "Measures"  is 
associated  with  and  is  derivative  of  the  applications.  A 
mathematical  formulation,  "Mathematics,"  appropriate  for 
analyses,  underlies  this  architecture. 

The  focus  of  the  Workshop  was  to  develop  a  framework  that  could 
be  used  for  evaluation  of  the  systems.  This  chapter  identifies 
this  structure  and  attempts  to  answer  the  questions:  "What  did  we 
accomplish?"  and  "What  do  we  still  need?" 
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3. 1  Architecture  for  Evaluations  -  What  Did  We  Accomplish? 


This  Workshop  was  one  of  a  continuing  number  of  occasional 
efforts  to  build  a  science  of  evaluation  using  "measures  of 
effectiveness."  Through  this  forum,  the  level  of  awareness  among 
analysts  and  decisionmakers  as  to  the  lack  of  consistency  and 
structure  was  raised.  Moreover,  perhaps  because  of  the  new  nature 
of  C^,  it  is  evident  that  very  little  work  is  being  done  to  correct 
these  deficiencies. 

We  believe  that  an  architecture  for  evaluation  such  as 
presented  here  is  most  needed  in  the  difficult  task  of  preparing 
comparable  measures  about  unlike  systems  which  are  competing 
alternatives  for  budget  resources. 

Figure  3-1  is  a  representation  of  this  structure.  The 
application  will  be  used  to  specify  the  scope  of  the  analysis.  A 
conceptual  model  would  further  specify  the  important  modules  for 
development  and  generation  of  MOEs  used  to  support  decisions  related 
to  the  Application.  Data  sources,  parameter  types,  and  mathematical 
formulations  would  follow.  The  ultimate  goal  in  a  specific 
evaluation  would  be  to  identify  the  mix  and  match  of  applications, 
boundary  conditions,  models,  measures  needed,  and  techniques  for 
data  collection.  Such  a  menu  approach  will  facilitate  the 
structuring  of  analyses. 

This  chapter  also  identifies  some  of  the  specific  contributions 
of  each  working  group  which  impact  on  the  architecture.  It  must  be 
emphasized  that  the  application,  rather  than  any  specific  model,  is 
the  driving  force.  The  application  is  the  blueprint  for  elaborating 
the  specifics  of  the  architecture  without  the  need  for  further 
analysis  in  making  specific  decisions  tied  to  the  Application.  Thus 
the  analysis  is  shaped  by  decisions  it  is  intended  to  support. 
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WHAT  IS  THE  STRUCTURE? 


FIGURE  3-1 

STRUCTURE  -  ARCHITECTURE 
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3.1.1  Applications 


The  specific  focus  of  measures  of  effectiveness  analyses  depend 
upon  the  objectives  of  the  study.  The  "Applications"  working  group 
identified  four  sets  of  decisionmaker  environments  objectives  which 
may  require  distinct  approaches:  concept  definition,  development, 
acquisition,  and  operations.  These  terms,  used  in  the  DOD  5000 
series,  describe  the  life  cycle  phase  of  military  systems.  Their 
inclusion  in  DOD  documentation  emphasizes  the  fact  that  these  are 
different  phases  where  decisions  that  DOD  personnel  must  make  occur, 
and  that  our  analyses  must  support. 

Individual  or  comparative  system  assessments  may  be  required 
within  these  Applications. 

In  an  individual  assessment,  we  ask:  "Can  we  determine  the 
contribution  of  a  specific  system  to  command  and  control  as 
exercised  by  military  commanders  or  mission  performance?" 

On  the  other  hand,  the  extreme  case  of  comparative  analyses  is 
reflected  in  the  question:  "Can  we  compare  all  DOD  physical 
systems,  from  bombs  to  computers,  across  all  Services,  commands,  and 
mission  areas?"  These  latter  questions  are  asked  in  various 
tradeoffs  such  as  those  employed  in  the  POM  cycle. 

The  specific  analytical  questions  being  addressed,  together 
with  the  applications  area  to  which  the  study  is  directed,  determine 
the  boundaries  to  be  used.  The  more  specific  the  tasking  statement, 
the  more  passive  the  boundary.  If  the  tasking  is  too  broadly 
stated,  boundaries  used  for  establishing  limitations  of  scope  of  the 
analysis  are  likely  to  be  too  encompassing,  making  the  analysis 
intractable  or  so  lacking  in  specifics  as  to  not  be  supportive  of 
decisionmaking.  This  approach  and  its  limitations  are  common  to 
systems  approaches  to  analysis  in  many  different  disciplines.  If 
the  tasking  is  overly  narrow  for  the  decision  being  supported, 
however,  answers  to  suboptimal  questions  are  derived. 


3-^ 


The  application  thus  directs  us  to  the  focus  of  the  analysis 

within  a  general  conceptual  model.  The  general  model  is  made 

specific  with  regard  to  physical  systems,  processes,  and  level  of 

2 

aggregation.  The  general  conceptual  model  of  C  is  viewed  as  the 
2 

foundation  for  C  system  evaluation. 

From  this  perspective,  the  tools  to  use  become  clear.  These 

tools  are  the  measures  and  the  mathematics.  Next,  we  will  discuss 
2 

the  C  system  model. 

3.1.2  Model 

For  much  of  this  Workshop  and  its  predecessor,  the  1984  C^I  MOE 

Symposium,  the  search  was  directed  toward  identifying  a  generic 

model.  "Generic,"  as  used  here,  refers  to  a  model  of  sufficient 

generality  to  be  applicable  regardless  of  mission  area.  Service, 

2 

command,  or  C  system.  Additionally,  the  model  must  accommodate  the 
2 

entire  C  system,  including  physical  entities,  structure,  and  its 
environment,  especially  weapons  systems.  Functionality  would  be  a 
prime  focus.  Time,  space,  geography,  and  hierarchical  relationships 
could  be  imbedded  in  or  in  the  environment  of  the  model.  Such  a 
model  should  be  capable  of  relating  any  of  its  components  to  overall 
force  effectiveness. 

2 

The  Workshop  defined  a  C  system  as  a  combination  of  structure 

2  p 

and  physical  equipment.  The  C  process  is  what  the  C  system 
does.  How  we  analyze  the  process  and  the  system  within  an  environ¬ 
ment,  as  well  as  what  aspects  we  specifically  study  and  the 
complexity  of  the  model,  are  functions  of  the  application. 

Unfortunately,  it  is  frequently  the  case  that  different  parts  of  our 
2 

C  community  evaluate  one  or  the  other  of  these  two  aspects,  rather 
than  both. 

p 

To  handle  either  individual  or  comparative  C  system 
assessments  in  an  inclusive  context,  a  detailed  model  may  be 
needed.  The  level  of  analytical  detail  depends  upon  such  factors  as 
the  criticality  of  the  decision,  the  time  available  for  the 
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analysis,  and  the  phase  of  the  system  life  cycle  which  the  model 
supports.  The  use  of  boundaries  substantially  reduces  the  size  and 
focus  of  the  model,  especially  with  highly  complex  models. 

2 

This  working  group  emphasized  one  part  of  the  C  system, 
namely,  the  decision  process.  This  perspective  is  based  upon  the 
assumption  that  a  major  function  of  control  systems  is  to  help 
make  better  decisions,  and  that  a  clear  statement  of  this 
perspective  is  badly  needed. 

Attempts  were  made  to  extend  the  resulting  C  decision  process 
model  to  hierarchical  systems,  two-sided  systems,  and  temporal 
considerations,  with  some  success,  indicating  potential  payoff  in 
the  next  iteration. 

3.1.3  Measures 

MOEs  and  MOFEs  are  the  basic  terms  for  representing 
"effectiveness."  The  precise  combination  of  measures  used  depends 
upon  the  analysis  objectives,  conceptual  model,  boundaries,  and  the 
nature  of  the  analysis.  The  application  determines  whether  to 
evaluate  the  force  effectiveness  or  simply  the  performance  of  a 
given  system.  The  level  of  the  analysis  impacts  upon  the 
specification  of  the  boundaries  for  the  model.  However,  there  are 
significant  problems  with  availability  of  the  data.  For  example,  no 
objective  data  represents  the  entire  generic  model.  Both  quantita¬ 
tive  and  qualitative  data  must  be  included.  Further,  depending  upon 
how  broadly  construed  the  analysis  objective  is,  we  must  recognize 
that  certain  data,  for  example,  in  the  Strategic  Defense  Initiatives 
arena,  is  unlikely  to  be  available  for  analysis  purposes.  Under 
such  circumstances,  simulations  and  other  techniques  could  provide 
substitute  essential  data.  Even  these  data  sources  may  not  be 
available,  depending  upon  the  time  frame  of  the  analysis  or  the  size 
of  the  problem.  Furthermore,  MOEs  and  MOFEs  typically  require 
analysis  of  two  opposing  forces  and  frequently  too  little  data,  if 
available,  about  the  enemy. 
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3.1.^  Mathematics 


The  other  major  tool  needed  for  the  evaluation  of  systems  is 
a  method  of  relating  the  inputs  to  a  set  of  outputs  that  allows 
individual  or  comparative  assessments.  The  working-group  members 
defined  two  essential  modeling  dimensions.  First,  they  developed  a 
probabilistic  formulation  which  reflects  uncertainty  and  tests  for 
model  completeness.  Second,  they  formulated  mathematical  components 
integrating  the  measures  of  dimensions,  effectiveness,  and  force 
effectiveness. 

3.2  What  Do  We  Still  Need? 

The  architecture  we  are  proposing  is  not  a  complete 
structure.  It  is  a  framework  to  provide  specific  direction.  The 
way  to  progress  is  to  accept  some  structure  that  has  been  developed 
within  the  profession,  test  it,  and  iterate  it.  For  example, 
operational  models  of  the  process  should  be  constructed.  Other 
parts  of  the  C  system  model  remain  to  be  developed  or 
synthesized.  Perhaps  the  many  good-to-excellent  models  available 
for  specific  purposes  might  be  incorporated  into  a  generic  model. 

However,  we  are  not  now  sufficiently  advanced  to  provide  this 
level  of  sophistication.  We  have  identified  scxne  of  the  issues  that 
relate  to  such  an  approach  and  some  of  the  areas  where  progress  is 
being  made.  Many  more  blanks  remain  to  be  filled.* 

We  are  proposing  to  use  the  structure  being  established  to 
generate  additional  analytical  blocks.  In  this  way,  different 
techniques  may  be  incorporated  into  an  architecture  for  the 

p 

evaluation  of  C  systems. 


A  few  of  these  have  been  filled  even  while  we  wrote  this  report. 
Specifically,  Chapter  8  indicates  the  evolutionary  nature  of  our 
conceptualization  of  the  evaluation  structure  about  which  this 
chapter  reports  the  June  1985  version. 
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This  architectural  strawman  must  be  applied  to  real-world 
problems.  The  applications  must  be  assessed.  If  they  provide 
realistic  solutions,  this  approach  will  be  validated.  If  the 
architecture  is  productive  for  these  applications,  we  should  test 
the  limits.  If  not,  we  need  to  determine  why,  and  where  it  needs  to 
be  modified.  Only  if  these  findings  are  exchanged  within  the 
community  will  we  continue  to  build  on  what  has  been  established  and 
accepted. 

The  exchange  of  experiences  has  already  begun  with  respect  to 
the  theory  of  a  generic  conceptual  model.  Claims  have  been  made 
that  no  general  model  is  needed.  Do  we  need  an  explicit  generic 
model  when  we  can  work  the  parts?  Do  we  need  a  theory  of  measures 
of  effectiveness  when  we  work  specific  analyses?  Yet,  more  and  more 
frequently,  the  demand  is  made  to  relate  specific  analyses  to 
general  force  effectiveness.  In  essence,  there  is  growing  demand 
for  an  explicit  generic  model.  Reality  tells  us  that  for  any 
specific  study  we  haven't  time  to  develop  a  generic  model  as  defined 
here.  Even  if  it  were  developed,  that  model  would  be  used 
modul arly. 

2 

The  architecture  is  designed  to  ensure  that  evaluation  of  C 
systems  is  based  upon  appropriate  factors.  The  architecture  should 
be  used  for  comparative  evaluation  of  alternative  systems  and  to 
assess  individual  systems.  The  impact  of  the  system  on  other 
components  of  the  model,  e.g.,  organizations,  processes,  time,  and 
space,  can  then  be  determined. 

The  characteristics  we  use  today,  e.g.,  technical  and  schedule 
uncertainty,  effectiveness,  impact  on  communications  and  other 
resources,  will  certainly  be  included.  Further,  new  variables  will 
be  added  relating  to  total  force  effectiveness,  arms  control,  and 
technological  progress.  Finally,  sensitivity  analysis  may  be  used 
to  estimate  the  ranges  of  conditions  and  to  determine  bounds  of 
uncertainties. 
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The  architecture  will  expand  as  concepts  are  refined.  A  set  of 
tools  will  be  developed.  These  tools  will  allow  the  analyst  and  the 
decisionmaker  to  specify  the  problem  quickly  and  succinctly,  and 
then  to  proceed  with  the  work  of  answering  the  specific  questions 
involved  in  evaluating  systems. 

3 .3  Existing  Building  Blocks 

Despite  a  number  of  remaining  unresolved  areas,  the  Workshop 

can  look  with  pride  at  several  significant  accomplishments.  First, 

the  structure  we've  called  the  architecture  for  the  evaluation  of 

systems  has  been  made  explicit.  Often,  prior  work  has  been  guided 

intuitively  in  determining  the  pieces  for  the  required  analysis,  but 

without  a  suitable  integrating  structure.  Moreover,  we've  firmly 

tied  the  "evaluation"  effort  to  the  specific  decisions  that  need  to 

be  made,  i.e.,  to  the  application  area  which  the  analysis  is 

designed  to  support.  These  decision  requirements  establish  the 

boundaries  of  the  model  and  the  scope  of  the  analysis.  In  addition, 

2 

the  nature  of  the  C  system  was  specified.  A  frequently  overlooked 

.  p 

major  component,  the  C  process,  was  conceptually  modeled. 

Two  opposing  views  of  the  need  for  a  generic  model  have 
surfaced.  Whenever  the  requirement  is  to  determine  the  force 
effectiveness  gain  attributable  to  a  system,  an  inclusive  generic 
model  is  applicable.  In  contrast,  other  analyses  focus  on  the 
subdivisions  of  the  generic  model,  and  more  commonly  smaller 
special-purpose  models  developed  for  each  problem.  This  behavior 
results  in  the  assertion  that  a  generic  model  may  not  be  needed. 

The  measures  selected  were  tailored  to  suit  the  applications 
and  the  analytical  model(s).  The  results  of  the  analysis  then  more 
explicitly  reflect  the  decisionmaking  needs. 

Finally,  the  mathematics  were  designed  to  accommodate  a  variety 
of  possible  models  to  enable  formulation  without  restriction  on  the 
model  contents.  This  effort  resulted  in  a  flexible  and  generic 
mathematical  framework  for  the  evaluation  of  systems. 
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3.4  Concluding  Remarks 

The  existing  foundations  of  this  effort  were  established  by 
decisionmakers  and  analysts  at  this  and  prior  Workshops  and 
Symposia.  We  have  developed  a  broad  structure  for  evaluation,  but 
existing  and  new  studies  must  be  integrated.  We  must  not  view 
current  inconsistency  as  indicative  of  lack  of  progress.  Only 
through  an  architectural  approach  to  evaluation  will  the  full 
potential  of  systems  be  realized.  In  order  to  obtain  maximum 
operational  effectiveness,  the  capability  must  be  understood.  And 
in  order  to  comprehend  both  the  objective  and  subjective  aspects,  it 
must  be  subjected  to  an  evaluation  procedure  that  provides  the 
breadth  needed  to  include  all  its  disparate  parts. 


3-1  0 


CHAPTER  li 


APPLICATIONS  AND  THE  NEED  FOR  C^  MEASURES 

by 

Dr.  William  Foster 
COL  Robert  Allison 
Robert  Choisser 
LtCol  Edward  C.  Jonson 
Dr.  S.  Z.  Mikhail 
MAJ  Bernard  Galing 
MAJ  Larry  Rhoads 


n.O  INTRODUCTION 

The  objective  of  this  chapter  is  to  discuss  applications  of  the 

2  2 
C  measures  and  process  model  to  the  analysis  of  C  systems  or  to 

p 

analysis  of  larger  systems  in  which  C  systems  may  be  imbedded. 
Referring  to  the  progress  in  this  regard,  Mr.  Charles  Zraket  stated 
at  the  198^1  Symposium  at  The  MITRE  Corporation: 

"We  have  still  not  succeeded  in  formulating  either  an 
analytic  methodology  or  a  systematic  evaluation  process 
to  deal  with  the  two-sided  dynamics  of  C^  in  warfare  in 
contrast  to  analyzing,  e.g.,  strategic  force  exchanges 
with  static  drawdown  curves." 

The  key  element  in  the  quoted  lament  is  the  absence  of  a  systematic 
evaluation  process.  Measures*  are  successfully  used  every  day  but 
in  an  ad  hoc  manner.  This  obstacle  that  has  prevented  the  complete 
success  of  previous  attempts  at  application  of  measures  must  be 
overcome. 

The  thesis  of  this  report  is  that  measures  have  wide 
application  in  both  conceptual  and  implementation  areas  (categories) 

P 

involved  in  the  design,  acquisition,  and  operations  of  C  systems. 


*The  term  "measures,"  used  throughout  Chapter  refers  to  areas  in 
vrfiich  Measures  of  Performance  or  Measures  or  Effectiveness  might  be 
used. 
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Measures  must  be  determined  through  an  analytic  effort  that  is 
as  quantitative  in  its  approach  as  possible.  However,  the  extent  of 
potential  quantifications  is  a  function  of  the  nature  of  the 
applications.  On  the  one  hand,  the  formulation  of  budgetary  POM 
decisions,  decisions  related  to  system  design,  the  acquisition 
process,  operational  concerns,  or  more  conceptual  applications, 
e.g.,  assessment  of  new  technology,  development  of  R&D  goals,  and 
the  determination  of  contribution  to  force  effectiveness,  are 
likely  to  show  greater  quantification.  On  the  other  hand,  the 
development  of  doctrine  or  generating  or  validating  requirements  is 
likely  to  be  more  qualitative.  Careful  review  is  needed  to  ensure 
the  appropriate  level  of  quantification  and  to  identify  areas  of 
potential  bias. 

The  extent  of  objectivity  will  be  evident  if  the  purpose  of  the 
analysis  is  set  forth.  The  supporting  analytical  model  must  be 
succinctly  described.  Data  collection  must  be  consistent  with  the 
model  and  be  performed  under  specified  conditions.  To  the  extent 
possible,  measurements  should  be  repeatable.  The  translation  of 
model  inputs  into  measures  of  the  system  which  are  amenable  to 
validation  should  be  transparent  for  the  decisionmaker  and  traceable 
for  the  analyst.  The  applicability  of  capability  assessment, 
tradeoff  analysis,  and  risk  analysis  to  the  conceptual  and 
implementation  categories  must  be  determined  in  the  analysis. 

4.1  Chapter  Organization 

This  chapter  contains  descriptions  of  the  appropriate  appli- 
cation  of  C  measures  for  analysis  in  conceptual  and  implementation 
categories,  defined  below.  The  following  sections  set  forth  appli- 
cation  considerations  based  upon  the  nature  of  the  C  system,  the 
environment  in  which  the  system  must  operate,  the  interrelationship 
of  this  system  with  other  systems,  and  other  special  aspects  that 
affect  its  development  or  operation. 
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Section  ^.2  defines  the  application  categories  and 
subcategories.  Analytical  implications  of  these  applications  are 
identified  in  Section  ^.3.  Guidelines  for  application  of  measures 
to  specific  categories  are  set  forth  in  Section  ^(.4.  Examples  of 
applications  are  given  in  Section  4.5  and  Section  4.6  presents 
conclusions  and  recommendations  of  the  working  group. 

2 

4.2  Pertinent  Areas  for  Application  of  C  Measures 

2 

There  are  many  areas  of  analysis  in  which  C  measures  can  be, 

2 

or  should  be,  incorporated.  These  range  from  the  analysis  of  C 

2 

equipment,  in  which  the  C  measures  may  be  the  only  effectiveness  or 

performance  gauges,  to  the  analysis  of  forces  engaged  in  battle,  in 
2 

which  C  measures  are  but  one  of  many  different  measures  used  to 
determine  force  effectiveness.  MOEs  and  MOFEs  are  related  to  both 
the  operational  context  and  the  boundary  of  the  C  system.  These 
are  derived  from  the  analysis  objectives.  It  seems  likely  that 
different  sets  of  generic  measures  are  required  for  different 
applications. 

2 

Since  the  scope  of  analysis  in  which  C  measures  should  be 

applied  varies  greatly,  it  is  useful  to  derive  a  set  of  broad 

analysis  categories.  These  categories  can  then  be  used  to  determine 

2 

how  best  to  apply  C  measures.  The  collection  of  analysis 

categories  which  follows  has  been  organized  into  two  distinct 

subcategories:  those  that  address  the  conceptual  examination  and 

2 

those  concerned  with  the  implementation  of  C  capabilities. 

Further,  this  section  provides  specific  guidelines  regarding  the 
types  of  measures  needed  for  each  application  category.  Using 
Table  4-1  as  the  framework,  some  of  the  special  requirements  and 
considerations  for  measures  that  apply  to  each  of  the  application 
categories  are  discussed. 


ELEMENTS  OF  APPLICATION 
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Tradeoff  Analysis 
Risk  Analysis 
Capability  Assessment 


^.2.1  Conceptual  Category 


The  conceptual  category  includes  those  areas  involving  the 

formulation  of  concepts  or  doctrine,  and  those  areas  relating  to  the 

planning  required  to  achieve  future  capabilities,  including  force 

requirements  and  force  objective  capabilities.  The  applications  in 

this  category  begin  at  the  higher  level  of  doctrine  development,  but 

2 

also  subsume  the  generation  and  validation  of  requirements  for  C 
systems  and  the  evaluation  of  the  contribution  these  systems  make  to 
force  effectiveness.  This  very  important  aspect  of  force  and  system 
evaluation  is  the  most  difficult  to  quantify  and  places  a  heavy 
burden  on  the  mathematical  formulation  and  modeling.  However,  it 
also  provides  a  basis  for  assessing  both  promising  new  technology 
and  research  and  development. 

^.2.1.1  Doctrine  Development.  These  analyses  examine  ideas  from 

the  grand  strategy  level  to  the  major  military  and  naval  commands. 

For  example,  the  Air-Land  Battle  concept  becomes  doctrine  through 

the  use  of  idea  generation,  subjective  analysis,  models, 

simulations,  war  games,  and  military  exercises.  measures  are  not 

often  directly  addressed  in  the  models  and  simulation  portions  of 

such  analyses,  but  are  frequently  included  in  the  more  narrowly 

focused  supporting  models  and  simulations.  Subjective  analyses,  war 
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games,  and  military  exercises  should  include  C  measures,  although 
they  will  often  not  be  quantifiable. 

Doctrine  alternatives  are  normally  evaluated  by  commanders  and 
operations  analysts.  Evaluation  of  doctrine  alternatives  should  be 
based  upon  tradeoff  and  risk  analyses.  In  these  analyses,  both  MOEs 
and  MOFEs  are  required.  The  same  set  of  MOEs/MOFEs  must  apply  to 
all  alternative  doctrines.  The  selected  set  must  reflect  the 
condidate  doctrine  with  respect  to  its  function  (mission)  within  the 
operational  environment.  As  a  minimum,  the  doctrine  must  be  related 
to  force  effectiveness  and  be  generic. 


4. 2. 1.2  Requirements  Generation  and  Validation  Process.  The 
Military  Services  and  the  Unified  and  Specified  Commands  have  formal 
procedures  to  generate  and  validate  requirements.  These  result  from 
the  official  recognition  that  some  new  capability  is  or  will  be 
needed.  An  approval  authority  will  validate  the  requirement  that  a 
new  capability  is  needed.  measures  are  frequently  needed  to 
analyze  stated  requirements  and  to  assist  in  their  clarification  and 
validation.  Several  examples  may  be  shown.  For  example,  C 
measures  may  be  used  to  produce  the  specifics  of  a  new  require¬ 
ment.  Credible  analyses,  generated  from  postulated  future  enemy 
capabilities  or  from  hierarchically  structured  C^  measures,  also  can 
be  used  in  deriving  specific  C^  requirements  from  much  broader 
objectives  statements.  C^  measures  are  appropriate  to  verify  that  a 
new  requirement  exists. 

The  generation  and  validation  of  operational  requirements  for 
systems  normally  involve  the  decisionmakers  listed  in  Table  4-1, 
subordinate  commanders,  and  analysts.  Engineers  are  usually 
restricted  to  technical  consultation.  Alternative  requirements  at 
the  mission  and  function  levels  are  evaluated  using  tradeoff 
analysis,  risk  analysis,  and  capability  assessment. 

Hierarchically  structured  MOEs  and  MOFEs  show  the  force 
effectiveness  impacts  of  each  alternative  requirements  level.  The 
requirements  establish  (1)  minimum  acceptable  values  for  the  MOPs, 

(2)  subsequent  verification  of  meeting  the  requirements,  and 

(3)  comparison  with  existing/programmed  capabilities  to  identify 
shortfalls. 

In  evaluating  systems  with  respect  to  requirements,  credit 
should  be  given  for  exceeding  a  requirement  by  viewing  it  as  a  lower 

bound. 


^•2.1.3  Evaluation  of  Contribution  to  Force  Effectiveness.  The 

2  2 
development  of  a  C  measure  to  determine  the  contribution  of  C  to 

force  effectiveness  is  urgently  needed.  Joint  and  combined  forces 

may  be  effective  either  through  deterrence  of  combat  or  through 

p 

success  in  combat.  Robustness  of  C  is  intuitively  a  strong  factor 
2 

in  deterrence.  C  analysis  measures  should  demonstrate  the 
contribution  of  to  the  avoidance  of  war. 

2 

The  contribution  of  C  to  force  effectiveness  or  to  success  in 
combat  may  be  analyzed  with  respect  to  small  forces,  or  to  joint  and 
combined  forces.  The  force  effectiveness  of  a  small  ground  force 
supported  by  close  air  support  having  a  variable  communications 
capability  with  the  forward  air  controller  would  require  both 
measures  and  weapons  effectiveness  measures.  For  larger  forces, 
such  as  combined  forces  on  a  theater  battlefield,  measures, 
particularly  those  pertaining  to  command  and  control,  should  be 
applied  to  aggregate  force  capabilities. 

2 

C  contribution  to  force  effectiveness  must  be  determined  using 
MOFEs  vidienever  a  "least-common  denominator"  is  required  for  compar¬ 
ing  disparate  types  of  systems,  e.g.,  a  sensor  system  and  a 
communications  system.  Since  MOFEs  are  scenario  dependent,  system 
evaluation  requires  using  several  different  scenarios.  The  specific 
values  of  MOFEs  are  largely  subjective,  so  data  from  field  exercises 
and  military  experience  may  be  used  to  provide  needed  insights.  For 
some  applications,  the  MOFEs  can  be  framed  within  a  single  military 

p 

mission.  However,  when  C  systems  span  several  mission  areas,  they 
must  be  related  to  the  ability  to  wage  a  particular  type  of  war, 
representing  the  range  of  all  pertinent  missions. 

^•2.1  .^4  New  Technology  Assessment.  New  technology  assessment  is 
performed  by  technical  experts  and  analysts  who  determine  the 
military  ramifications  of  the  projected  technical  capabilities. 
Tradeoff  analysis,  risk  analysis,  and  capability  assessment  evaluate 
the  relative  merits  of  C  systems  employing  the  new  technologies  to 


those  employing  existing  technology.  MOFEs  are  required  to  evaluate 
the  capability  to  conduct  the  applicable  military  mission  using  the 
proposed  technologies.  Technical  risk  will  be  a  particularly 
important  factor.  MOPs  reflecting  the  forecast  technical 
capabilities  will  be  related  to  the  MOEs  and  MOFEs  through  a 
hierarchical  structure. 

11.2.1.5  Setting  Research  and  Development  Goals.  The  determination 
of  which  program  to  be  pursued  is  closely  related  to  both  "new 
technology  assessment"  and  "requirements  generation  and  validation" 
conceptual  application  areas.  Since  alternative  technologies 
compete  for  resources,  measures  are  useful  to  illustrate  either 
their  absolute  or  relative  values. 

R&D  goals,  established  to  remedy  identified  operational 
shortfalls  of  present/programmed  capabilities,  are  determined  by 
operations  analysts,  systems  engineers,  and  technical  experts. 

The  shortfalls  and  alternative  means  of  mitigating  tiiem  are 
identified  using  tradeoff  analysis,  risk  analysis,  and  capability 
assessment.  MOEs  will  be  required  vdien  goals  must  be  prioritized, 
necessitating  a  common  measure  of  the  utility  of  disparate  goals. 
MOPs  must  be  related  to  required  candidate  system  capabilities  and 
the  specific  technical  improvements. 

i|.2.2  Implementation  Category 

The  implementation  category  is  more  quantifiable  than  the 
conceptual  category.  This  category  includes  measures  to  assist  in 
the  development  planning  and  tradeoffs  of  programs  from  early  in  the 
Program  Objectives  Memorandum  (POM)  and  budget  cycle  through  the 
acquisition  process  and  into  the  technical  and  operational 
evaluation  of  these  systems.  Tnere  is  a  danger  that  the  implemen¬ 
tation  category  may  be  overemphasized,  merely  because  it  is  more 
amenable  to  quantification  and  is  less  abstract  in  nature. 
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Attention  must  be  paid  to  the  structure  of  the  system  as  well  as  the 
physical  entitles  in  this  category  to  the  extent  possible. 

The  implementation  phase  begins  by  the  selection  of  a  program 
from  a  set  of  competing  programs,  each  defined  by  a  set  of  accepted 
requirements.  Once  a  program  is  selected,  the  normal  system 
development  cycle  is  followed,  namely:  selection  of  a  specific 
design,  construction  of  an  advanced  development  model  and  then  an 
engineering  development  model,  conduct  of  technical  and  operational 
evaluations  of  the  engineering  development  model,  and,  if 
successful,  implementation  of  full-scale  production. 

^•2.2.1  POM/Budget  Cycle.  All  levels  of  the  funds  justification 

2 

Cham  need  applicable  C  measures.  At  a  very  low  level,  i.e.,  at  a 
2 

base,  a  C  measure  might  be  employed  in  analyzing  the  operational 
effectiveness  improvement  brought  on  by  replacing  an  existing 
telephone  switching  system.  At  higher  levels,  e.g..  Department  of 
the  Army,  an  application  example  of  measures  might  be  the 
investigation  of  the  benefit  of  adding  capabilities  to  satellite 
systems. 

The  evaluation  of  alternative  programs  for  funding  normally 
involves  executives  from  the  system  commands  as  well  as  military 
commanders.  System  command  executives  could  be  analysts, 
scientists,  or  engineers.  The  merits  of  each  program,  the  set  of 
requirements  each  program  fulfills,  and  its  contribution  to  the 
effectiveness  of  the  force  which  it  is  designed  to  support  should  be 
addressed  by  tradeoff  analysis  and  risk  analysis.  MOEs  and  MOFEs 
are  required  for  all  candidate  programs.  If  different  capabilities 
as  indicated  by  MOPs  are  shown  with  similar  MOE/MOFE  values,  then 
additional  MOEs  should  be  developed. 

^•2.2.2  Acquisition  Process.  Analyses  supporting  the  acquisition 
process  are  focused  upon  design,  development,  production,  and 
field ing/ implementing  of  systems.  Applicable  measures  include 
measures  of  hardware  and  software,  and  measures  which  reflect  the 


training,  operational  procedures,  transition  to  new  systems,  and  the 

introduction  of  new  maintenance  inventories.  The  effectiveness  of 

systems  critical  to  national  security  during  transition  is  not  a 

trivial  problem.  Another  measurement  requirement  relates  to 

2 

including  the  "human-in-the-loop"  aspects  of  C  in  the  analysis. 

The  objective  of  the  acquisition  process  is  to  select  one  of 
the  candidate  system  architectures/designs  and  carry  it  through  to 
the  mass-production  phase.  The  evaluation  of  candidate 
architectures/designs  of  a  specific  system  normally  involves 
military  commanders  and  program  managers  from  system  commands, 
engineers,  analysts,  scientists,  and  military  commanders.  Because 
of  the  pervasive  nature  of  some  C  capabilities,  a  large  number  of 
interfaces  and  interoperability  considerations  must  be  included  in  a 
valid  analysis.  Each  alternative  should  be  addressed  with  tradeoff 
analysis,  risk  analysis,  and  capability  assessment  in  terms  of  one 
set  of  MOEs/MOFEs.  Life-cycle  costs  and  development-cycle  time  must 
also  be  emphasized.  Requirements  should  be  expressed  in  terms  of 
minimum-acceptable  values  for  each  MOE.  The  selected  MOPs  and  MOEs 
must  be  suitable  for  measuring  the  relative  ability  of  the 
candidates  to  fulfill  the  requirements. 

ii.2.2.3  Technical  Evaluation.  Technical  evaluation  determines  how 

well  existing  or  developmental  hardware/software  systems  meet  their 

2 

design  criteria.  Performance  measures  must  be  tailored  to  the  C 

2 

system  under  investigation.  For  larger  systems,  of  which  C  is  only 
one  of  several  components,  C  measures,  measures  of  other  system 
components,  e.g.,  communications  and  weapons  systems,  and  possibly  a 
relationship  from  these  measures  to  an  overall  system  MOFE  may  be 
used. 

The  technical  evaluation  of  a  system  will  determine  the 
degree  to  which  a  selected  hardware  configuration,  e.g.,  the 
engineering  development  model,  fulfills  the  specifications.  It  is 
normally  conducted  by  the  system  developer  under  the  supervision  of 
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system  command  executives  and  the  program  manager.  An  Operational 
Test  and  Evaluation  Command  is  normally  represented. 

Tradeoff  analysis,  risk  analysis,  and  capability  assessment 
will  usually  be  needed  to  address  alternatives  for  further 
development,  testing,  and  operational  implementation.  The  results 
of  a  technical  evaluation  should  be  expressed  in  the  same  measures 
adopted  when  selecting  the  system  architecture/design. 

^■2.2.4  Operational  Evaluation.  Operational  analysis  combines  the 
technical  evaluations  with  the  organizational  dimension.  Thus,  the 
analysis  evaluates  the  ability  of  existing  or  developmental 
organizations,  with  their  hardware/software,  to  meet  the  objectives 
for  which  they  were  created.  The  emphasis  in  these  analyses  is  as 
much  on  the  C  aspects  as  it  is  on  communications. 

Operational  evaluation  is  usually  performed  as  one  of  the  last 
activities  in  the  development  phase  before  mass  production  of  a 
system  is  allowed.  Passing  an  operational  evaluation  is  usually 
necessary  for  the  acceptance  of  a  system  by  the  responsible  Service. 

The  potential  utility  of  the  system  is  determined  using 
tradeoff  analysis,  risk  analysis,  and  capability  assessment. 

Force  commanders,  special  test  organizations  unique  to  each 
Service,  system  command  representatives,  and  system  development 
agencies  take  part  in  the  operational  evaluation  process. 

MOFEs,  MOEs,  and  MOPs  are  applicable  to  operational  evaluation 
with  emphasis  on  MOFEs,  which  are  the  primary  output  measures  in 
this  case.  The  MOFE/MOE/MOP  set  used  in  operational  evaluation 
should  be  identical  or  at  least  closely  analogous  to  those  used  in 
architecture/design  selection  and  comprise  a  superset  of  those  used 
in  technical  evaluation. 

4.3  Application  Considerations 

This  section  outlines  the  analytical  implications  of  each 

O 

general  application  area  for  C  measures.  These  analytical 


implications  include:  application  and  decomposition  (or  structure), 
bounding  conditions  and  caveats,  translation  of  needs  and  results 
between  and  among  decisionmakers  and  analysts,  differences  in  focus 
to  account  for  current  and  future  capabilities,  and  (5)  observations 
about  the  practicalities  of  application.  They  are  discussed  for 
both  conceptual  and  implementation  applications. 

The  analysis  approaches  and  techniques  must  (1  )  be  tailored 
to  the  analysis  objective,  (2)  consider  alternatives  and  risk 
reduction,  (3)  take  the  system  operational  environment  into  account, 
and  (^)  include  command  structure  and  force  employment  options 
(orders  of  battle). 

ij.a.l  Application  and  Structure  of  Analysis 

Table  ^1-1  indicates  the  type  of  analysis  appropriate  for  each 
application  area,  together  with  the  structure  of  analysis  elements 
and  the  level  at  which  decisions  are  made.  These  vary  and  are 
unique  for  conceptual  as  compared  to  implementation  applications. 

In  the  conceptual  arena,  which  is  largely  mission-building/ 
decomposition,  for  example,  in  doctrine  development  and  objectives, 
e.g.,  national  and  Service  level,  are  the  starting  point.  Next, 
various  levels  of  strategy  to  implement  tiiese  objectives  must  be 
defined.  Then  missions  and  employment  must  be  specified. 

Thereafter,  Service  doctrine  must  encompass  these  elements. 

In  implementation,  the  primary  system  is  decomposed,  doctrine 
is  defined  and  goals  are  set.  For  example,  in  programming  and 
planning  the  POM/budget,  the  analysis  will  likely  start  the 
decomposition  at  the  mission  area  and  then  descend  through  program 
elements  to  programs  or  projects.  This  breakdown  is  useful  to 
understand  and  support  the  building  of  the  DOD  (Service,  Component) 
investment  strategy. 
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It  should  be  noted  that  the  type  of  decisionmaker  varies  with 
the  kind  of  application  area.  In  the  conceptual  area,  the 
decisionmaker/commander  dominates  the  final  decision  with  support 
from  planners,  programmers,  and  developers.  In  the  implementation 
area,  planners,  programmers,  and  developers  dominate  decisionmaking 
in  the  planning  and  acquisition  processes. 

^.3.1.1  Analysis  Versus  Decisions.  It  is  suggested  that  a 
decision-oriented  analysis  approach  be  taken.  This  approach  must 
include: 

a.  Mission  goals  and  objectives. 

b.  Decisions  required  to  achieve  the  goals  and  objectives. 

c.  Information  needed  to  make  the  required  decisions.  From 
this  information,  problem  definition  for  analysis  begins, 
leading  to  the  solution  of  questions  such  as:  (1)  Do 
current  capabilities  satisfy  the  mission  requirement? 

(2)  Will  programmed  capabilities  satisfy  the  mission 
requirement?  (3)  What  modifications  (process  or  system), 
new  capabilities,  or  new  technology  are  needed?  (^1)  What 
operational  alternatives  are  viable? 

To  develop  a  decision-oriented  analysis  approach,  one  must  also 
focus  on  the  intended  use  and  the  likely  user.  Thus  information 
needs  may  be  expanded  to  identify  the  information  to  decisionmaking 
and  the  information  to  be  produced  as  a  result  of  the  decision 
outputs.  Identifying  information  needs  must  be  related  to  the 
decision  level,  vfliich  relates  the  required  data  to  decisionmaker 
needs  and  is  essential  to  achieve  practical  results  to  properly 
support  decisionmaking.  Table  H-2  portrays  both  the  decisionmaker 
and  user  organizations  based  upon  application  area  and  analysis 
objectives. 

^.3.1.2  Use  of  Analytical  Tools.  Three  broad  analytical  tools  are 
useful  in  these  applications,  namely,  capability  assessment,  risk 
analysis,  and  tradeoff  analysis.  Within  each  application  area,  the 
analysis  objective( s) ,  the  portrayal  of  results,  and  me  level  at 
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which  the  decision  is  made  roust  be  well  defined.  These  tools  are 
related  to  their  use,  indicating  special  considerations  in  Table  4-3. 

The  saroe  types  of  analysis  are  used  in  different  application 
areas.  However,  the  thrust  of  the  analysis  and  the  questions  to  be 
answered  roust  be  uniquely  defined  for  each  problem.  Further,  the 
approach  must  consider  the  information  required  by  the  decisionmaker 
for  that  application. 

4.3.2  Practicalities  of  Application 

For  most  efficient  use,  the  analysis  must  be  carefully 
structured,  taking  into  account  the  realities  of  decisionmaking. 

Table  4-4  outlines  practical  considerations.  The  primary  factors 
describe  some  of  the  bounding  constraints  of  the  analysis.  The 
observations  characterize  the  likely  nature  of  the  analysis,  e.g., 
more  qualitative  than  quantitative,  as  well  as  provide  insight  into 
capability,  development,  planning,  and  programming  needs.  It  is 
stating  the  obvious  to  note  that  implementation  work  will  tend  to  be 
more  quantitative  and  better  defined  than  conceptual  work. 

4.4  Examples  of  MOE  Application 

4.4.1  MOEs  in  the  TACAMO  Capability  Analysis 

Evolving  technologies,  both  of  the  United  States  (U.S.)  and  the 
Union  of  Soviet  Socialist  Republics,  led  to  the  conclusion  several 
years  ago  that  a  major  program  would  have  to  be  initiated  by  the 
late  1980s  to  increase  the  U.S.  capability  to  command  and  control 
its  undersea  missile  fleet.  Preliminary  analyses  reduced  the 
options  for  consideration  to  two — either  to  procure  more  C-130s,  or 
to  buy  a  replacement  aircraft,  the  E-6. 

The  primary  analytical  requirement  was  to  meet  essential 
needs  at  a  minimum  total  cost.  The  overall  MOE  was,  in  general 
terms,  "How  well  does  this  system  (either  the  C-130  or  the  E-6)  meet 
the  C  control  criteria?"  The  requirement  was  expressed  as  the 
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SELECTED  APPLICATIONS  VERSUS  TOOLS 
(CONSIDERATIONS) 
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Operational  How  does  the  system  (equipment,  Need  normal  (Intended)  logistic 

Evaluation  procedures,  manpower)  perform  support.  Must  use  planned  operational 

In  the  Intended  operational  procedures • 

environment? 


need  to  provide  an  .xx  probability  of  reaching  yyj^  of  the  submarines 
in  the  required  time.  Soviet  jamming  power  and  nuclear  bursts 
Intended  to  disrupt  were  assumed.  The  driving  parameters  of  the 
analysis  were  the  strength  of  electronic  signals  emitted  by  the 
aircraft  and  aircraft  location  in  relation  to  a  geographically  fixed 
array  of  submarines. 

It  is  interesting  to  note  that  aircraft  speed  and  endurance 
(time  aircraft  can  remain  aloft),  vdiich  have  nothing  to  do  directly 
with  communications,  were  actually  MOPs  by  virtue  of  their 
relation  to  the  overall  MOEs.  These  factors  drove  the  estimate 
of  how  many  aircraft  were  required.  MOPs  more  commonly  understood 
as  directly  related  to  communications  included  communications 
equipment  reliability,  signal  strength  (including  the  power- 
generation  capability  aboard  the  aircraft),  and  message 
transmission/retransmission  accuracy . 

MOEs  in  Redeye  Employment 

The  Digital  Communications  Terminal  (DCT)  was  used  in  early 
1978  as  a  device  to  assist  the  Redeye  gunner  in  the  employment  of 
his  weapon.  The  DCT  was  connected  to  the  Tactical  Air  Operations 
Center,  vhich  provided  air-traffic  information.  The  Redeye  gunner 
now  had  a  display  which  provided  a  much-increased  range  for 
observation  and  indication  of  the  type  of  aircraft,  i.e.,  friendly 
or  enemy. 

Operational  tests  were  conducted  to  determine  if  a  gunner  could 
effectively  use  the  DCT  and  the  information  provided  to  engage 
targets  successfully.  In  order  to  remove  the  gunner's  inherent 
ability  to  observe  and  then  engage  a  target,  he  was  placed  under  a 
poncho,  which  limited  his  vision  to  only  the  DCT  and  Redeye 
displays.  The  gunner  was  required  to  observe  the  indications  from 
the  DCT,  point  his  weapon  in  the  proper  direction  indicated  by  the 
DCT  to  detect  an  ememy  aircraft,  and  then  once  the  weapon  indicated 
acquisition  of  ti%&  aircraft,  fire  on  the  aircraft  within  the 
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engagement  envelope  of  the  missile.  This  activity  was  expected  to 
be  accomplished  without  any  direct  visual  observation  of  the 
aircraft  by  the  gunner. 

The  DCT  provided  a  map  display  of  the  gunner's  position  and 
selected  key  terrain  around  his  position.  The  Tactical  Air 
Operations  Center  provided  the  location  and  classification  of 
targets  to  the  DCT  for  display  on  the  map.  The  target  information 
was  provided  in  near  real~time  potentially  sufficient  for 
acquisition. 

C^  measures  were  applied  to  this  test  at  two  levels,  with  a 
third  not  considered.  The  first  level  was  an  MOP  which  considered 
the  ability  of  the  gunner  to  read  and  understand  the  data  provided 
by  the  map  to  produce  useful  information  for  his  mission.  The 
second-level  measure,  an  MOE,  was  the  evaluation  of  the  application 
of  the  information  provided  in  successful  engagement  of  targets. 

The  third-level  measure,  an  MOFE,  not  addressed  during  this  test  but 
important  to  employment  of  the  system,  was  the  contribution  of  the 
DCT  to  enhance  the  effectiveness  of  Redeye  as  a  close-in  air-defense 
system. 

4.M.3  Selection  of  an  Architecture  for  POD  Common-User 
Telecommunications 

This  example  is  taken  from  a  recent  effort  to  select  the 
preferred  alternative  for  the  Worldwide  Digital  System  Architecture 
(WWDSA).  The  purpose  of  the  architecture  was  to  serve  as  an  overall 
"umbrella"  for  the  development  of  all  future  DOD  common-user 
telecommunications  systems.  The  alternatives  were  developed  by 
first  establishing  the  user  requirements  and  comparing  them  with  the 
existing  capability  to  determine  needed  improvements,  and  then 
postulating  a  set  of  significantly  different  architectures  for 
providing  the  needed  improvements.  The  selection  process,  which  is 
the  focus  of  this  example,  involved  development  of  a  hierarchical 
set  of  effectiveness  and  performance  measures,  along  with  a  set  of 
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measures  reflecting  the  various  types  of  implementation  penalties, 
scoring  the  alternatives  against  each  measure,  and  deciding  which 
alternative  is  "best"  in  an  overall  sense. 

The  top-level  measures  are  shown  in  Table  4-5.  Note  the  basic 
dichotomy  between  "effectiveness,"  which  measures  all  benefits 
expected  to  accrue  from  having  a  particular  architectural 
alternative,  and  "implementation,"  which  measures  all  penalties  that 
must  be  paid  for  employing  it. 

Effectiveness  was  decomposed,  at  the  next  level,  into  four 
categories.  "Capability"  measures  how  well  the  system  does  its 
basic  task  of  transferring  information  when  the  network  is 
undamaged.  "Survivability"  measures  the  resistance  of  the  system  to 
sustaining  damage  from  enemy  attack,  along  with  its  ability  to 
perform  in  a  partially  damaged  state  and  its  ability  to  restore  some 
of  its  destroyed  capabilities.  "Adaptability"  measures  how  well  the 
system  can  respond  to  changing  environmental  conditions,  e.g.,  its 
ability  to  extend  its  boundaries,  reconfigure  its  connectivity, 
accommodate  traffic  peaks,  and  interoperate  with  other  systems. 
"Security,"  which  measures  the  integrity  of  the  secure 
communications  service,  could  logically  have  been  placed  under 
"Capability,"  which  was  subdivided  to  highlight  this  vital 
consideration. 

No  method  presently  exists  for  relating  a  common-user 
telecommunications  system  to  force  effectiveness,  since  the  system 
serves  a  multiplicity  of  user  types,  both  for  primary  and  backup 
connectivity,  and  must  be  "all  things  to  all  people."  Accordingly, 
the  highest  level  of  effectiveness  reflected  how  well  the  system 
accomplished  its  basic  job  of  moving  information,  without  regard  to 
how  the  receipt  of  this  information  might  have  an  impact  on  force 
effectiveness.  The  development  of  a  general  method  for  measuring 
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TABLE  4-5 


WWDSA  TOP-LEVEL  MEASURES 


Effectiveness 

•  Capability 

•  Security 

•  Survivability 

•  Adaptability 

Implementation 

•  Cost 

-  Life  Cycle  Cost 

-  R&D 

-  Implementation 

-  O&M 

-  Manning 

-  Spectrum 

•  Risk 

-  Technical 

-  Cost 

-  Schedule,  etc. 

•  Technical 

•  Cost 

•  Schedule 

•  Transitioning 

-  Uniformity  of  Cost  Profile 

-  Continuity  of  Effectiveness 
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the  contribution  of  common-user  telecommunications  systems  to  force 
effectiveness  is  an  important  challenge  facing  the  C  community. 

Table  M-6  shows  the  successive  decomposition  of  the 
"Capability"  category,  as  an  illustration  of  the  tree  structure. 
Similar  structures  were  developed  for  the  other  three  effectiveness 
categories. 

No  further  decomposition  of  "Implementation"  was  used  beyond 
that  shown  in  Table  k-5.  "Cost"  refers  to  all  valuable  resources 
that  must  be  expended  to  realize  an  architecture,  which  accounts  for 
the  inclusion  of  manning  and  spectrum  under  this  category.  The 
three  subdivisions  under  "Risk"  pertain  to  uncertainties  in  being 
able  to  achieve  the  assumed  values  with  respect  to  effectiveness, 
cost,  and  schedule,  respectively.  Finally,  "Transitioning"  refers 
to  the  relative  ease  with  which  the  present  systems  can  evolve  into 
the  architecture  under  evaluation,  as  measured  by  the  flatness  of 
the  required  funding  profile  and  the  lack  of  degradation  in 
effectiveness  during  the  transition  period. 

Explicit  definitions  were  constructed  for  the  lower  level 
criteria  in  order  to  promote  a  common  understanding  of  the 
quantities  being  scored.  The  criteria  developed  for  WWDSA  should  be 
generally  applicable  for  any  common-user  telecommunications  system 
if  modified  as  appropriate  to  highlight  different  design  factors  and 
to  extend  down  to  lower  levels. 

The  scoring  of  alternatives  against  the  measures  was  conducted 
by  a  team  of  experts  collectively  knowledgeable  in  all  aspects  of 
the  problem.  The  scores  were  expressed  in  terms  of  relative  utility 
and  were  assigned  initially  at  the  bottom  level  of  the  tree.  These 
scores  were  propagated  to  the  top  level  of  effectiveness  through  the 
assignment  of  importance  weights  by  means  of  a  multiattribute 
utility  model.  The  Effectiveness  scores  were  plotted  against  the 
Implementation  scores  to  show  dominance  of  alternatives  and  the 
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TABLE  4-6 


Timeliness 


Quality 


Service 


-Circuit  Service 
(Voice/Data/ 
Hard  Copy) 


Transaction  Service 
(Data/ Hard  Copy) 


Clear  Voice 


Secure  Voice 


Data,  Hard  Copy  - 

Common/ Dedicated  User 
Ratio 

Switch  Features 


Special  Services 


•Call  Completion  Rate 
Call  Interrupt  Rate 


Call  Placement  Time 


■Speed  of  Service 


Intelligibility 

Naturalness 

Intelligibility 


Naturalness 


Error  Rate 


Selection  Offered 


•Effectiveness 

Offered 


"knee  of  the  curve."  These  scores  were  also  combined  through  the 
use  of  relative  weights  to  yield  an  overall  "figure  of  merit." 
Numerous  sensitivity  excursions  were  made  through  computerization  of 
the  scoring  process. 

The  decisionmakers  were  pleased  with  the  results  and  felt 
confident  that  the  best  alternative  had,  in  fact,  been  selected. 

4.5  Conclusions  and  Recommendations 

In  this  chapter,  the  application  of  MOEs  based  upon  the 

analysis  objectives  has  been  addressed.  The  specification  of  MOEs 

is  treated  in  other  chapters.  After  the  the  perusal  of  these 

chapters,  the  reader  may  have  a  better  view  of  both  the  availability 

2 

and  adequacy  of  tools  for  evaluating  C  systems. 

An  important  aspect  of  OE  application  is  the  clear  delineation 
of  the  caveats  related  to  the  formulation  and  use  of  these 
measures.  The  process  must  be  as  objective  as  possible  and 
perspective  must  be  maintained  so  that  it  can  be  recognized  if  the 
model,  the  analysis,  or  the  MOE  applies  to  the  system  being 
addressed.  As  MOEs  are  applied  to  the  areas  addressed  in  this 
chapter  (including  considerations  (Section  ^.3)  and  guidelines 
(Section  4.4)),  the  adequacy  of  the  MOEs  as  a  tool  must  be  assessed 
and  fed  into  the  process. 

4.5.1  MOE  Application  -  The  Process 

As  a  process,  the  formulation  of  MOEs  is  recursive  and 
iterative.  As  illustrated  in  Table  4~1 ,  the  determination  of 
shortfalls  in  MOEs  applied  to  a  system  is  fed  back  into  the 
conceptual  model  so  that  this  model  can  be  modified,  refined,  or 
changed.  Over  time,  the  MOEs  can  become  accurate  measures  of  an 
existing  system  or  can  be  modified  to  cope  with  evolutionary  changes 
in  the  system,  the  environment,  or  the  scenario. 


4-24 


4.5.2  Realities  of  the  Process 


The  application  of  MOEs  must  be  reviewed  as  a  process  in  order 
to  ascertain  the  fidelity  of  results  to  reality.  Some  aspects  of 
this  determination  have  been  discussed  above.  The  review  should 
answer  questions  similar  to  the  following: 

a.  Do  the  MOEs  appear  to  match  the  system  being  evaluated?  Do 
these  measures  derive  from  an  appropriate  model?  Is  the 
data  consistent?  If  surprises  exist,  can  they  be 
explained? 

b.  Was  the  data  accurate,  reliable,  and  relevant?  Does  it 
appear  sufficiently  complete  to  apply  the  MOEs? 

c.  In  collecting  and  assembling  data,  were  the  methods  of  data 
reduction,  the  standards  against  which  they  were  measured, 
and  the  method  of  dealing  with  "gaps"  or  "holes"  acceptable 
and  credible? 

d.  How  was  the  data  reduced?  How  much  information  or 
dimensionality  was  lost  in  the  data  collection  and 
reduction  processes? 

e.  Were  subjective  techniques,  if  used,  based  on  sound 
Judgment?  Were  they  sufficiently  explained  to  Justify 
their  contribution  to  the  analysis?  Was  their  relevance  to 
the  conclusions  affected  by  these  techniques? 

f.  Looking  at  the  analysis,  to  what  changes  in  parameters  are 
the  conclusions  and  findings  sensitive?  Is  the  sensitivity 
analysis  relevant  and  credible? 

p 

g.  From  the  viewpoint  of  the  C  system  program  manager, 
designer,  or  operator,  how  significant  are  the  conclusions 
and  findings  derived  from  the  application  of  the  MOEs? 

Have  we  verified  vhat  we  already  know  (status  quo),  or  does 
this  analysis  contribute  to  programmatic,  design,  or 
operational  decisions? 

4.5.3  "Selling"  the  Analysis 

If  MOEs  are  expected  to  be  useful,  they  must  be  accepted  by 

decisionmakers.  For  a  considerable  length  of  time,  results  of  MOEs 

2 

derived  for  the  analysis  of  C  systems  have  failed  to  convince 
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decisionmakers,  whether  they  are  operational  commanders,  commanders 

of  system  commands,  DOD  officials,  or  members  of  the  Executive 

Branch,  e.g..  Office  of  Management  and  Budget,  or  the  Congress. 

Care  must  be  taken  to  apply  principles  successfully  used  to  "market" 

2 

systems  and  to  relate  MOE  analysis  to  the  capability  of  C 
systems.  Transparent  models  and  simple  MOEs  are  helpful.  However, 
the  character  and  characteristics  of  the  audience  must  be  considered 
in  preparing  presentations,  briefings,  or  reports.  When  subjective 
techniques  or  judgments  are  necessary  during  the  analysis,  they 
should  be  considered  frankly  and  honestly.  If  "soft"  areas  appear 
in  the  data  or  the  analysis,  the  conclusions  should  identify  a 
caveat  so  that  the  integrity  and  credibility  are  discernible  to  the 
audience.  A  coherent  story  line  should  extend  from  assumptions 
through  the  analysis  to  the  conclusions.  Finally,  conclusions 
should  be  clearly  linked  to  the  analysis  presented. 
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5.0  INTRODUCTION 

This  chapter  introduces  and  develops  the  conceptual  model  of  a 
C^  process  and  relates  it  to  C^  systems  for  the  purpose  of  measuring 
various  system  designs.  Also  presented  is  an  explicit  description 
of  the  process  with  a  discussion  of  how  it  can  be  applied  to 

p 

analyzing  C  systems. 

5.1  Background 

2 

Chapter  2  provides  the  definition  of  C  and  extends  it  to 

p  2 

introduce  the  concept  of  C  systems  as  well  as  the  C  process.  The 

2 

distinction  between  these  two  concepts  is  that  the  C  system 
represents  the  physical  entities  and  the  structure  of  what  is  needed 
to  perform  C  ,  while  the  C  process  represents  the  C  functions  of 
how  C  is  performed. 

Most  military  analysts  have  an  intuitive  understanding  of  these 

p 

concepts  and  how  they  relate  to  their  C  systems.  However,  there  is 
not  sufficient  agreement  on  any  general  model  of  the  "what"  and 
"how"  of  these  systems  for  purposes  of  comparative  analysis  and, 
ultimately,  decisions  about  their  procurement  and  use.  The  reasons 
for  this  lack  of  consensus  are  many  but  they  can  generally  be 
reduced  to  three.  First,  most  analysts  believe  that  the  uniqueness 
of  each  Service's  role  and  mission  defies  generalization.  Second, 

p 

they  also  believe  that  C  is  so  complex  that  each  analysis  requires 
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a  fresh  and  unique  approach  depending  upon  the  system  and  situation 
being  evaluated.  Finally,  the  results  of  all  analyses  are  usually 
special  purpose. 

Consequently,  models  are  not  readily  communicated  among 
different  users  or  adaptable  for  other  purposes,  e.g.,  results  of 
analyses  for  procurement  of  systems  may  not  be  applicable  to 
evaluation  of  doctrine.  Accordingly,  a  fundamental  representation 

p 

of  the  processes  inherent  in  any  C  system  must  account  for  these 
concerns  if  it  is  expected  to  be  useful  as  a  tool  for  communicating 
across  a  wide  variety  of  analytic  concerns. 

5.2  Requirements  for  the  Model 

2 

The  following  requirements  must  be  met  if  a  conceptual  C 
process  model  is  to  be  considered  complete; 

a.  The  model  must  be  understood  and  agreed  to  by  all  types  of 
users  for  both  conceptual  and  implementation  applications. 

b.  The  model  must  represent  all  the  functions  addressed  by  a 

system  as  well  as  serve  as  the  basic  building  b^ock  for 
analysis  of  individual  process  components  of  the  C  system. 

c.  The  model  must  have  the  ability  to  clearly  define  the  ^ 
boundary  of  the  system  by  differentiating  between 
components  and  non-C  force  components. 

d.  The  model  must  allow  for  representation  of  time  and 
organization  among  and  within  individual  C  entities. 

e.  The  model  must  provide  the  ability  to  represent  the 
internal  dynamics  of  the  system,  such  as  iterative  and 
reflexive  (short-circuited)  processes  described  below,  as 
well  as  the  interactions  of  the  system  with  the 
environment. 

2 

f.  The  model  must  provide  the  framework  for  measuring  the  C 
system  at  three  levels,  in  particular,  MOPs,  MOEs,  and 
MOFEs. 

g.  The  model  must  consider  human  decisionmakers  from  the 
standpoint  of  cognitive  processing  factors. 
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h.  The  model  must  provide  the  ability  to  represent  information 
transfer. 

5.3  Assumptions 

To  develop  a  model  that  satisfies  the  above  requiranents,  it  is 
necessary  to  make  several  assumptions: 

a.  A  definable,  fundamental  process  exists  that 
comprehensively  describes  the  functional  aspects  of  the 
system. 

b.  This  process  is  a  functional  description  equally  applicable 
to  the  simplest  and  the  most  complex  systems. 

c.  This  process  or  combinations  of  this  process  can  be 
analyzed  in  context,  provided  a  goal  statement  and 
environment  can  be  articulated. 

5.4  Process  Model 

This  section  presents  the  basic  model  and  demonstrates  its 

ability  to  be  combined  to  provide  a  hierarchical  representation  of 
2 

military  C  structure.  Also  described  is  the  applicability  of  the 
model  to  a  conventional  timeline  analysis.  Next,  the  techniques  are 
extended  to  a  two-sided  model  which  is  representative  of  in  a 
Red/Blue  engagement.  The  model  is  then  subjected  to  a  simplified 
timeline  analysis.  Finally,  the  model  is  applied  to  super¬ 
ordinate  and  subordinate  systems. 

5.4.1  The  Basic  Model 

The  primary  intent  of  the  Conceptual  Model  working  group  was  to 
provide  a  basic  model  that,  in  its  most  simple  form,  represents  the 
basic  constituents  of  the  most  simple  process  and  yet  can  be 
extended  to  the  most  complex  system.  The  basic  model  is  shown  in 
Figure  5-1.  As  shown,  there  are  two  interactions  with  the 
environment.  These  interactions  are  represented  by  a  stimulus  input 
and  a  response  output.  The  output  can  only  cause  an  action  through 
our  own  forces,  which  results  in  a  change  to  the  overall 
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FIGURE  5-1 

CONCEPTUAL  C2  PROCESS  MODEL 


environment.  External  inputs  are  shown  coming  from  the  environment 
and  are  susceptible  to  both  natural  and  human- initiated 
environmental  effects.  The  only  other  direct  input  to  the  process, 
the  Desired  State,  establishes  an  error  function  inside  the  loop. 
This  error  function  causes  processing  activity  to  continue,  or,  at 
the  other  extreme,  halts  processing  activity  when  the  Desired  State 
is  believed  to  be  reached. 

Definitions  of  individual  functional  boxes  in  the  loop  are 
presented  below.  The  loop  process  is  initiated  when  a  sensed  input 
is  assessed  and  is  determined  or  is  believed  to  be  in  error  with  a 
Desired  State  or  when  our  requirements  for  the  Desired  State 
change.  These  errors  cause  the  generation  and  selection  of  options, 
which  result  in  a  plan  intended  to  change  the  environment.  The 
objective  is  to  minimize  the  difference  between  the  assessed  and 
desired  environment. 

5* ^*1.1  Definition  of  Functions.  This  section  provides  definitions 
of  the  functional  blocks  of  the  process  model  shown  in 
Figure  5-1 s 

a.  Sense  -  That  function  which  collects  data  necessary  to 
describe  and  forecast  the  environment,  which  includes: 

1.  The  enemy  forces'  disposition  and  actions. 

2.  The  friendly  forces'  disposition. 

3.  Those  aspects  of  the  environment  that  are  common  to 
both  forces,  e.g.,  weather,  terrain,  and  neutrals. 

b.  Assess  -  That  function  which  transforms  data  from  the  Sense 
function  into  information  about  intentions  and  capabilities 
of  enemy  forces  and  about  capabilities  of  friendly  forces 
for  the  purpose  of  determining  if  deviation  from  the 
Desired  State  warrants  further  action. 

Generate  -  That  function  which  develops  alternative  courses 
of  action  to  correct  deviations  from  the  Desired  State. 
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d.  Select  -  That  function  which  selects  a  preferred 
alternative  from  among  the  available  options.  It  includes 
evaluation  of  each  option  in  terms  of  criteria  necessary  to 
achieve  the  Desired  State. 

e.  Plan  -  That  function  which  develops  implementation  details 
necessary  to  execute  the  selected  course  of  action. 

f.  Direct  -  That  function  which  distributes  decisions  to  the 
forces  charged  with  execution  of  the  decision. 

5.i|.1.2  Implicit  Considerations  in  the  Structure  of  the  Conceptual 
Model.  The  basic  model  has  a  very  simple  form,  with  all  of  the 
functions  performed  sequentially  by  the  conceptual  C  system.  A 
real  system,  however,  may  execute  the  functions  in  a  distributed 
way,  and  substantial  interactions  may  exist  among  sets  of  C  systems 
in  the  performance  of  the  functions.  Further,  a  given  system  may, 
at  times,  appear  to  omit  some  of  the  functions  or  have  loops  within 
the  model  that  allow  the  execution  of  the  basic  functions  in  a 
different  order. 

These  apparent  variations  can  actually  be  accommodated  within 
the  conceptual  model.  However,  some  of  the  functions  may  be 
performed  implicitly.  For  example,  a  reflex  action  may  appear  to  be 
produced  by  the  simpler  model  shown  in  Figure  5-2 . 

All  the  functions  in  the  process  are  actually  performed,  though 
some  may  not  be  consciously  recognized.  A  reflex  action  that 
directs  an  action  from  sensed  data,  for  example,  results  from  a 
previously  learned  response.  That  is,  the  system  that  performs 
reflexively  has  learned  to  execute  the  process  in  a  way  that 
implicitly  and  rapidly: 

a.  Assesses  the  situation. 

b.  Generates  the  alternatives. 

c.  Selects  the  appropriate  alternative. 
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STIMULUS 


FIGURE  5-2 

SIMPLER  REFLEX  FORM  OF  CONCEPTUAL  C2  PROCESS  MODEL 
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The  conceptual  model  can  also  be  applied  to  more  complex 
systems  in  which  the  process  is  executed  in  a  distributed  way  or 
where  multiple  systems  interact.  These  applications  are  discussed 
in  the  following  paragraphs. 

5. it. 2  Application  of  Model  to  Hierarchical  Systems 

Multiple  entities  (Figure  5-3)  can  be  represented  with 
combinations  of  the  fundamental  process.  This  example  shows  the 
coordination  of  two  subordinate  entities  operating  within  a  shared 
environment.  entities  expanded  to  represent  these  nodes  are  also 
shown  in  Figure  5-^.  Layered  coordination  functions  are  depicted  to 
illustrate  the  multiple  simultaneous  functions  that  can  occur 
independently  within  nodes  of  the  same  system,  e.g.,  two  brigades 
are  connected  to  a  third.  Coordinating  directives  are  provided  as  a 
Desired  State  condition  input  to  the  Assess  function  of  the 
independently  directed  elements.  This  Desired  State  condition  is 
the  output  of  the  Direct  function  of  the  coordinating  element. 

It  is  significant  to  note  that  this  is  a  functional 
representation  and  that  in  a  physical  system  the  Desired  State  may 
arrive  from  the  environment  via  the  Sense  function.  Also  implied  is 
the  premise  that  physical  limitations  may  cause  distribution  of 
functions.  Application  of  the  model  in  this  way  represents  the 
hierarchical  relationship  within  a  command  structure.  The  model  can 
apply  equally  well  to  the  coordination  of  activities  between 
parallel  nodes,  where,  for  a  specific  situation,  one  of  the  nodes 
may  "coordinate." 

A  similar  structure  can  also  describe  the  interactions  of 
different  processes  within  a  single  node.  Adaptive  radar  and 
communication  systems,  for  example,  will  require  processing  within 
the  Sense  function  to  distinguish  and  validate  (Assess) 
communications  messages  and  environmental  situations,  e.g.,  must  be 
able  to  differentiate  among  targets  and  communications. 
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DESIRED 

STATE 


FIGURE  5-3 

COORDINATION  OF  MULTIPLE  C2  ENTITIES 
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FIGURE  5-4 

C2  ENTITIES  EXPANDED  TO  PROVIDE  COOPERATION 
AMONG  INDEPENDENT  C2  NODES 


5.4.3  Two-Sided  Model 


Because  the  definition  of  the  basic  functions  within  the 
conceptual  model  does  not  specify  the  methods  that  will  be  used  to 
accomplish  the  functions,  the  structure  of  the  conceptual  model  for 
interactions  between  two  opposing  systems  is  very  similar  to  the 
model  for  two  cooperating  systems.  A  schematic  of  the  former  is 
shown  in  Figure  5-5.  As  with  the  cooperating  systems,  the  primary 
interactions  between  the  two  are  accomplished  through  the 
environment,  though  the  interactions  are  somewhat  indirect,  i.e., 
there  is  no  direct  communication  of  the  Desired  State  from  one  of 
the  systems  to  the  other,  and  each  desires  to  create  a  different 
environment. 

The  principal  differences  between  the  interactions  of  opposing 
versus  cooperating  systems  would  be  (1)  the  relationship  between  the 
Desired  States  (what  is  good  for  one  side  is  bad  for  the  other), 

(b)  the  treatment  of  uncertainties  in  assessment,  and  (c)  the 
specific  details  of  the  approaches  used  to  generate  systems  and 
select  the  course  of  action. 

5.4.4  Consideration  of  Time 

The  conceptual  model  does  not  explicitly  represent  time.  The 

p 

timing  and  sequencing  of  the  execution  of  the  functions  by  a  C 
system,  however,  are  major  characteristics  of  the  system. 

Examination  of  the  timeline  characteristic  of  the  system  has  long 
been  recognized  as  important,  especially  under  circumstances  in 
which  the  functions  are  performed  by  distributed  systems.  This  type 
of  analysis  is  especially  important  for  the  common  situations  for 
which  the  Desired  State  is  time  dependent  or  it  is  assessed  that  the 
deviation  of  the  actual  situation  from  the  Desired  State  is  time 
rel ated. 
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SIDE  1 

DESIRED 

STATE 


SIDE  2 

DESIRED 

STATE 


FIGURE  5-5 
TWO-SIDED  MODEL 
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An  example  of  the  relationship  between  the  functions  of  the 

process  and  a  timeline  is  demonstrated  in  Figure  5-6.  A  fundamental 

requirement  of  a  C  system  is  its  ability  to  effect  a  desired  change 

in  the  environment  within  an  induced  timeline  before  other  changes 

nullify  the  desired  change.  For  example,  if  "destruction  of  enemy 

aircraft  at  their  bases"  is  a  Desired  State,  then  the  system 

could  be  deemed  ineffective  for  the  particular  task  if  the  time 

2 

required  to  perform  the  C  functions  and  execute  the  selected  course 
of  action  exceeds  the  time  remaining  before  enemy  aircraft  are 
launched  from  their  bases. 

The  modeling  of  the  performance  of  the  functions  by  a 
distributed  system  on  a  timeline  becomes  extremely  conplex.  The  use 
of  a  timeline  model,  however,  can  help  in  sorting  out  the 
complexities  and  allocating  priorities  for  further  development  of 
the  capabilities  of  the  system  to  execute  the  functions.  Figure  5-7 
provides  an  example  of  a  schematic  timeline  model  of  a  distributed 
system  with  three  specialized  components:  a  commander,  an  analyst, 
and  a  sensor.  Each  of  the  components  could  be  modeled  as  a 
"system."  For  example,  a  radar  "sensor"  performs  all  of  the 
functions  in  the  C  process  in  order  to  accomplish  its  sensing 
mission.  In  a  higher  level  system,  however,  the  components  could  be 
viewed  as  specializing  in  the  performance  of  one  or  more  of  the 
functions,  and  some  portions  of  the  functions  may  be  executed  in 
parallel.  The  commander  and  the  analyst,  for  example,  may  begin  to 
assess  while  the  sensor  is  still  sensing.  Additional  time  can  be 
saved  for  the  activities  in  support  to  each  of  the  functions  by 
performing  some  of  the  activities  in  parallel.  The  incremental 
value  of  any  changes  in  the  way  that  the  system  performs  the 
functions  can  then  be  evaluated  in  terms  of  the  effects  on  the 
overall  timeline  of  the  entire  system. 
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EVENT  DETECTED 


FIGURE  5-6 

CONCEPTUAL  C2  TIMELINE 
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Ts  ACTION  IMPLEMENTED 


5.4.5  Application  to  the  System 

As  stated  in  the  previous  section,  the  conceptual  model  was 
shown  to  be  applicable  to  more  complex  systems.  In  that  section,  a 
hierarchical  system  was  presented  which  can  be  extended  to  a  more 
canplex  form  required  to  represent  military  structures.  This  is 
possible  by  emphasizing  certain  characteristics  of  these  structures; 
namely,  they: 

a.  Contain  several  parallel  paths  with  similar  subordinate 
structures. 

b.  Require  extensive  coordination  between  parallel  entities. 

A  simplified  application  of  the  conceptual  model  to  a 
representative  military  subordinate  problem  is  shown  in 
Figure  5-8.  For  decision  purposes,  the  Desired  State  of  the 
separate  entities  of  the  system  is  set  with  the  transmission  of  the 
Air  Tasking  Order  (ATO).  The  separate  subordinate  activities 
contained  in  the  parallel  structure  have  responsibility  for 
detecting  and  launching  attack  aircraft  to  engage  detected  enemy 
aircraft.  The  stimulus  to  the  Air  Display  Unit  results  in  sensed 
enemy  aircraft  and  a  directed  response  which  flows  horizontally  to 
the  operations  center.  This  response  now  becomes  the  stimulus  to 
the  Sense  functional  component  of  the  aircraft  operations  center 
process,  resulting  in  a  response  causing  launch  of  aircraft.  This 
launch  now  causes  the  change  in  the  overall  air  situation 
environment. 

The  same  basic  reasoning  will  allow  treatment  of  superordinate 
,  As  before,  the  Desired  State  is  established  by  the 
transmission  of  the  ATO.  The  engagement  process  covered  in  the 
above  section  is  still  valid.  The  sensed  activity/response  takes 
place  until  the  Desired  State  is  achieved.  At  this  point,  the 
results  of  the  subordinates  in  achieving  the  Desired  State  are 
reported  in  terms  of  response  to  the  superordinate  level. 
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AIR  DEFENSE  UNIT 


FIGURE  5-8 

SUBORDINATE  AND  SUPERORDINATE  MODEL  APPLICATIONS 
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5.5  Measuring  Systems  Using  the  Process  Model 


2 

The  process  model  can  be  used  to  clarify  the  part  of  the  C 
system  that  needs  to  be  evaluated  as  well  as  to  relate  the  three 
types  of  measures  introduced  in  Chapter  2  to  the  process  model. 

5.6  Selecting  the  System 

All  Services  now  have  systems  that  extend  from  the 
Commander-in-Chief  to  the  junior  rifleman,  airman,  or  seaman. 

However,  it  is  not  necessary  to  consider  the  entirety  of  these 
systems  in  all  evaluations.  Instead,  most  analyses  need  to  focus  on 
only  selected  portions  of  these  systems.  To  borrow  from  Enthoven 
and  Smith,  "How  Much  is  Enough?"  is  the  principle  when  selecting  a 
portion  of  the  systems  for  analysis.  For  example,  when  studying 
the  Army’s  Firefighter*  system,  how  much  of  the  total  systems 
must  be  included  to  sufficiently  answer  questions  regarding  the 
performance  of  Firefinder,  relative  to  the  performance  of  the  entire 
fire  support  system  and  of  the  force  as  a  whole? 

The  process  model  can  assist  in  defining  the  portion  of  the 
total  system  that  needs  to  be  analyzed  in  the  following  manner. 
First,  Firefinder  performs  a  Sense  function  for  the  C  system.  If 
measures  are  to  be  used  only  to  describe  the  technical  performance 
or  details  of  the  Firefinder  system  or  to  compare  them  to  known 
capabilities  of  like  systems,  then  no  other  entity  of  the  system 
need  be  included.  However,  if  it  is  desired  to  ascertain  how 
Firefinder  contributes  to  the  overall  fire  support  system,  then  the 
entities  that  perform  the  Assess,  Generate,  Select,  Plan,  and 
Direct  functions  associated  with  Firefinder  must  also  be  included  as 
part  of  the  system.  In  this  particular  example  the  functions 
relating  to  positioning  the  system  in  a  tactical  environment  as  well 


*Counterbattery/countermortar  radar. 
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as  transferring  its  sensor  output  into  firing  commands  for 
particular  weapons  systems  should  be  included. 

It  should  be  clear  that  an  analyst  can  use  this  approach  as  a 
guide  and  a  check  to  see  if  that  portion  of  the  system  being 
evaluated  is  necessary  and  sufficient  to  answer  the  questions. 

2 

5.7  C  Measures 

In  Chapter  2,  the  concept  of  three  levels  of  measuronent  for 
were  introduced.  This  concept  is  readily  extended  to  the  process 
model  developed  above. 

First,  MOPs  are  used  to  measure  how  well  a  particular  function 
2 

of  the  C  process  model  is  performed.  Reverting  to  the  Firefinder 
example  presented  earlier,  MOPs  would  be  used  to  describe  how  well 
Firefinder  Senses,  e.g.,  detection  range,  location  accuracy,  speed 
of  service. 

Second,  MOEs  measure  the  integration  of  all  functions  of  the 
process  model.  Again  referring  to  the  Firefinder  example,  any 
measure  of  effectiveness  must  include  not  only  the  Sense  function 
but  its  interaction  with  the  other  five  functions  as  they  relate  to 
the  selected  C  system.  For  example,  one  measure  of  how  well 
counter  battery  fire  is  commanded  is  "the  time  fron  enemy  fire  of  an 
artillery  piece  until  an  order  is  given  for  counterfire."  This 
measure  includes,  of  course,  not  only  how  accurate  and  fast  the 
Firefinder  system  determines  and  detects  the  enmy  firing  location, 
but  also  how  this  target  data  is  Assessed  in  terms  of  threat  and 
value  to  the  force  mission,  how  options  are  Generated  to  respond  to 
the  enemy  fire  (artillery  or  air  strike)  and  Selected,  Planned  (fire 
order  or  close  air  support  strike  is  prepared)  and  Directed 
(fragmentary  order  issued). 

Finally,  MOFEs  relate  the  system  to  the  force,  including 
weapons  capability.  A  measure  of  the  time  from  enemy  firing  of  an 
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artillery  piece  to  the  silencing  of  that  piece  would  be  an  MOFE. 

The  appropriateness,  completion,  and  timeliness  of  any  such  action 
are  further  examples.  For  example,  in  the  Firefinder  paradigm,  any 
system  that  only  generates  air  strikes  in  response  to  enemy 
artillery  fire  would  generally  yield  untimely  and  inappropriate 
responses  to  squelch  that  fire  because  delays  in  aircraft  arrival 
are  usually  long  unless  the  aircraft  was  loitering  in  the  area. 

5.8  Summary 

In  summary,  we  have  discussed  the  assumptions  and  requirements 

2 

necessary  for  a  fundamental  representation  of  a  generic  C  process 

2 

model.  Working  from  these  requirements,  the  C  process  model  is 
defined  to  consist  of  a  sequential  arrangement  of  the  six 
functions:  Sense,  Assess,  Generate,  Select,  Plan,  and  Direct. 

These  functions  operate  upon  and  within  a  defined  environment. 

Action  within  the  process  loop  is  initiated  by  a  perceived 
divergence  from  a  Desired  State  and  a  sensed  environmental  state. 

The  process  model  was  subjected  to  experiential  checking 
during  the  working-group  deliberations.  The  checking  was  designed 
to  exhibit  the  model's  adaptive  properties,  while  maintaining  its 
fundamental  structure.  These  checks  indicate  that  the  model  allows 
for  the  representation  of  multidimensional  processes  interacting 
internally  and  through  the  environment. 

Simple  timeline  analysis  examples  demonstrated  the  ability  to 
expand  or  collapse  the  model  functionally  while  maintaining  desired 
time  sequencing. 

5.9  Strengths/Weaknesses 

Significant  strengths  derive  directly  from  the  simplicity  of 
the  model  and  its  elementary  properties.  processes  can  be  easily 
visualized  as  an  ordered  arrangement  of  the  functional  blocks. 

These  ordered  arrangements  lend  themselves  to  mathematical 
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modeling.  Complex  systems  may  be  selectively  decomposed  using  a 
window  approach  and  constructs  of  the  basic  elements. 

The  use  of  lumped  functional  qualities  within  the  basic  model 
will  not  completely  describe  a  distributed  process  system.  The 
group  did  not  attempt  dynamic  representation  of  such  a  system; 
however,  discussion  indicates  that  a  time  sequence  stop-frame 
approach  may  be  fruitful. 

Open  questions  remain  concerning  positioning  the  functional 
blocks  of  the  model  within  the  environment  and  determining  the  exact 
entry  point  for  the  Desired  State  stimulus. 
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CHAPTER  6 


MEASURES 

by 

Richard  Miller 
Dr.  Harold  Glazer 
Linda  Hill 
Charles  Smith 
CAPT  Bruce  Thieman 


6.0  INTRODUCTION 

Previous  chapters  have  developed  a  framework  for  the  analysis 

2 

of  C  systems.  The  current  chapter  addresses  the  analysis  process, 
focusing  on  the  specification  and  selection  of  measures  of 
performance  and  effectiveness. 

6 . 1  The  Analysis  Process 


In  any  analysis  supporting  decisions  during  the  life  cycle  of  a 
system,  it  is  essential  to  be  able  to  relate  the  contribution  of  the 
various  alternatives  under  consideration  to  the  desired  objectives 
of  the  system,  or  military  force.  The  mechanism  by  which  this 
relationship  is  established  is  referred  to  as  the  "analysis 
process."  While  there  is  no  single  universally  accepted  structure 
that  defines  this  process,  it  is  generally  agreed  to  involve  a  set 
of  activities  similar  to  those  identified  in  Figure  6-1  and  listed 
below; 

a-  Problem  Formulation  involves  the  development  of  a  clear, 
well-defined  statement  of  the  issue  or  question.  As  a 
result,  appropriate  analysis  objectives  and  assumptions  are 
identified,  and  the  criteria  for  selecting  preferred 
solutions  are  selected.  These  criteria  are  the  measures  of 
performance  and  effectiveness  previously  defined  in 
Chapter  2. 

b.  Search  involves  the  identification  and  selection  of  a  set 
of  alternative  solutions  to  be  evaluated,  as  well  as  the 
means  to  develop  the  information  needed  to  conduct  a 
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FIGURE  6-1 

THE  ANALYSIS  PROCESS 
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comparison  of  those  alternatives.  The  means  alluded  to 
here  generally  refers  to  mathematical  models,  simulations, 
war  games,  experiments  or  tests,  and  their  associated  input 
data  requirements. 

c.  Comparison  involves  the  rank  ordering  of  the  alternatives 
under  consideration  on  the  basis  of  the  criteria  (MOP  and 
MOE/MOFE)  identified  during  problem  formulation. 

d.  Interpretation  involves  assessing  the  results  of  the 
analysis  in  terms  of  the  objectives  previously  selected. 

It  draws  on  the  information  and  insights  developed  thus  far 
by  the  analyst,  and  frequently  requires  the  use  of  judgment 
and  intuition.  Interpretation  usually  results  in 
conclusions  and,  in  some  instances,  proposed  courses  of 
action.  As  indicated  in  Figure  6-1 ,  interpretation  can 
(and  frequently  does)  result  in  looping  back  to  some 
previous  point  in  the  process,  e.g.,  to  Search,  to  refine 
some  aspect  of  that  analysis,  such  as  re-defining  some 
alternative  on  the  basis  of  insights  developed  during  the 
comparison  phase  of  the  analysis. 

e.  Verification.  Ideally  one  would  always  like  to  verify  the 
results  (interpretations)  of  an  analysis.  Unfortunately, 
particularly  in  military  analysis,  this  is  seldom  possible, 
especially  during  the  early  stages  of  the  system  life 
cycle.  As  an  alternative  to  verification,  analysts  and 
decisionmakers  must  generally  be  content  with  quality 
assurance  that  relies  heavily  on  peer  review  and  similar 
assessments  of  the  logic,  models,  and  data  employed  in  an 
analysis. 

The  proper  selection  of  the  criteria  to  be  used  in  comparing 
alternatives  is  one  of  the  most  important  steps  in  developing  an 
analysis  plan.  Furthermore,  the  ability  (or  inability)  to  specify 
the  values  of  those  criteria  will  eventually  influence  the  efficacy 
of  the  analysis  to  accomplish  the  objectives  defined  during  problem 
formulation.  Currently,  the  criteria  selection  process  is  more  an 
art  than  a  science,  and  may  always  be  so.  A  major  objective  of  this 
report,  however,  is  to  establish  a  common  basis  for  a  more 
structured  approach  to  this  selection  process.  Accordingly,  the 
major  focus  of  the  remainder  of  this  chapter  will  be  on  the 
selection  and  specification  of  the  system  and  force-level 
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measurement  criteria,  the  MOPs,  MOEs,  and  MOFEs  defined  in 
Chapter  2 . 

p 

6.2  C  Systems  Analysis 

The  previous  section  provided  a  general  discussion  of  the 
analysis  process  and  the  role  of  measures  (criteria)  selection  and 
specification  in  that  process.  This  section  will  focus  specifically 
on  the  analysis  of  systems.  The  goal  of  this  section  is  to 
develop  further  the  relationship  between  analysis  measures  and  the 
means  by  which  those  measures  are  specified,  and  the  purpose  and 
objectives  of  the  analysis.  At  any  point  during  the  life  of  a 
system,  certain  issues  are  of  particular  concern  to  system 
designers,  resource  managers,  and/or  users.  These  issues  determine 
the  objectives  and  focus  (or  scope)  of  the  analysis,  which  in  turn 
directly  influence  the  selection  of  appropriate  measures  to  be  used 
and  ultimately  the  means  of  specifying  those  measures. 

6.2.1  System  Life  Cycle 

The  life  cycle  of  a  system  has  traditionally  been  segmented 
into  three  phases:  concept  definition,  acquisition,  and 
operation.  The  objectives  of  each  life-cycle  phase  are  described 
below. 

6. 2. 1.1  Concept  Definition  Phase.  Develop  the  total  system  and 
program  requirements  from  a  broad  system  or  mission  objective.  The 
requirements  can  then  be  used  to  support  the  technical  and 
management  decisions  regarding  development  of  the  system.  During 
the  concept  definition  phase,  the  system  effectiveness  analysis  is 
used  to  develop  and  define  a  cost-effective  system  that  will  satisfy 
the  operational  mission.  The  characteristics  of  the  system  that 
most  affect  mission  objectives  are  identified.  Implications  of 
tradeoffs  between  the  desired  system  characteristics  and  other 
constraints  are  determined  so  that  realistic  goals  for  the 
characteristics  can  be  set. 


6. 2. 1.2  Acquisition  Phase.  Accomplish  detailed  engineering 
designs.  The  system  is  built  and  tested  to  determine  vdiether  or  not 
the  system  and  its  subsystems  meet  specifications.  Engineering 
reviews  and  test  programs  are  conducted  to  determine  contractor 
compliance  with  the  specifications. 

6. 2. 1.3  Operation  Phase.  Install  the  system  and  operate  and  test 
the  system  in  a  realistic  operational  environment.  Assess  the 
capability  of  the  system  to  meet  the  effectiveness  requirements 
originally  defined  in  the  concept  definition  phase.  During  the 
operation,  system  effectiveness  measures  may  be  used  to  reconfigure 
and  manage  an  operational  system  in  real  time  in  order  to  optimize 
resources  to  maximize  the  system's  effectiveness  or  performance. 

6.2.2  Types  of  Analyses 

Depending  on  the  nature  of  the  issues,  or  objectives  of  the 
analysis,  the  scope,  or  focus,  of  that  analysis  may  be  at  the 
subsystem,  system,  or  mission  level. 

6. 2. 2.1  Subsystem  Analyses.  Subsystem  analyses  are  limited  to  some 
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element,  or  component,  of  the  C  system  of  interest.  An  example  of 

such  an  analysis  would  be  an  evaluation  of  the  performance  of  radar 

2 

operating  as  part  of  a  larger  air-defense  C  system.  The  inputs  to 
such  an  analysis  are  dimensional  parameters.  The  output  criteria 
used  in  the  comparison  of  such  an  analysis  are  called  MOPs. 

6. 2. 2. 2  System  Analyses.  System  analyses  involve  assessments  of 
the  system.  Depending  on  the  scope  of  the  analysis,  the  inputs 
may  be  subsystem  dimensional  parameters  (in  the  case  of  very 
detailed  studies)  or  subsystem  MOPs  (in  more  aggregate  studies).  In 
any  event,  the  output  measures  are  system  MOEs. 

6. 2. 2. 3  Mission  Analyses.  Mission  analyses  are  designed  to  address 

the  contribution  of  the  system  to  the  military  force  of  which  it 

2 

is  a  part.  Mission  analyses  may  be  used  to  determine  the  C 
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system's  required  operational  capabilities  for  the  force  to 
accomplish  its  assigned  mission.  Also,  mission  analyses  may  be  used 
to  measure  the  contribution  the  system  makes  to  force  missions.  The 
inputs  to  such  an  analysis  may  be  subsystem-dimensional  parameters 
or  MOPs,  or  system  MOPs  or  MOEs,  depending  on  the  detail  required. 
The  output  measures  employed  are  mission-MOEs  or  MOFEs. 

6.2.3  Selection  of  Measures 

As  previously  mentioned,  the  phase  of  the  life  cycle,  the 
analytic  objectives,  and  the  analytic  focus  combine  to  determine  the 
measures  to  be  used  and  the  means,  i.e.,  models,  to  include  tests 
and  war  games,  to  be  employed  in  specifying  those  measures.  Tables 
6-1,  6-2,  and  6-3  illustrate  these  relationships  in  considerably 
more  detail.  Each  table  addresses  a  type  of  analysis;  mission, 
system,  or  subsystem.  For  each  type  of  analysis,  sample  measures 
and  model  scope  are  related  to  analysis  objectives  and  phases  of  the 
life  cycle.  Model  scope,  which  addressed  the  means  by  which 
measures  are  specified,  is  designed  to  summarize  the  extent  to  which 
certain  variables,  e.g.,  forces,  environment,  alternative 
descriptions,  tactics,  should  be  represented  within  those  means. 

6. 2. 3.1  Table  6-1  (Mission  Analysis).  The  measures  for  mission 

analysis  are  usually  MOFEs.  The  measures  chosen  must  provide  a 

quantitative  value  to  the  effects  of  the  system  upon  the  ability 

of  the  force/C^  system  to  carry  out  its  mission.  Thus  the  types  of 

measures  chosen  for  mission  analysis  are  directly  related  to  the 

outcome  or  force  status  during  and  at  the  culmination  of  the 

2 

particular  scenario  chosen  to  test  the  C  system. 

The  measures  can  be  very  generic,  such  as,  "Did  the  U.S.  win, 
lose,  or  draw?"  But  the  measures  could  be  more  specific,  such  as 
number  of  targets  destroyed  by  each  side  or  force  status  of  each 
side  during  and  after  the  scenario. 
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OBJECTIVES,  MODELS,  AND  SAMPLE  MEASURES  FOR  C‘^  MISSION  ANALYSIS 
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OBJECTIVES,  MODELS,  AND  SAMPLE  MEASURES  FOR  C‘^  SYSTEM  ANALYSIS 
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The  scope  of  the  model  for  performing  a  mission  analysis  must 
be  broad,  that  is,  it  must  have  a  representation  of  all  or  many  of 
the  components  of  the  friendly  and  enemy  force/C  structures, 
including  environmental  and  threat  effects,  tactics,  and  different 
strategies,  the  different  components  must  be  interactive  so  that 
the  effects  of  each  friendly  or  enemy  system  or  tactic  can  be 
determined  in  a  quantitative  manner. 

In  the  concept  definition  phase,  the  model  can  be  very  general, 
that  is,  representative  of  a  system  but  not  very  detailed.  In  the 
acquisition  or  operation  phases,  the  model  should  be  quite  realistic 
and  hence  very  detailed  in  its  complexity. 

6. 2.3. 2  Tables  6-2  and  6-3  (System  and  Subsystem  Analyses).  The 
best  measures  for  system  or  subsystem  analyses  may  be  MOEs  or  MOPs 
respectively.  The  measures  chosen  must  provide  a  quantitative  value 
for  the  ability  of  a  particular  system  or  subsystem  to  meet 
specifications  or  reasonable  performance  standards.  The  types  of 
measures  chosen  must  Indicate  the  quality  of  the  system's 
performance  to  the  decisionmaker . 

The  measures  can  be  somewhat  general,  such  as  the  survivability 
of  the  system,  or  very  specific,  such  as  the  probability  of 
detection  for  a  sensor  system  or  the  reliability  of  a  sensor 
subsystem  component. 

For  the  concept  definition  phase,  a  model  capable  of  closed- 
loop  operation  is  required.  Closed  loop  means  that  the  model  can 
interact  with  other  factors,  such  as  support  systems,  enemy  threat, 
environmental  effects,  and  the  forces.  However,  these  other  factors 
need  not  be  represented  in  detail  since  we  are  only  attempting  to 
determine  the  principal  characteristics  of  the  system  being 
considered  for  development. 


6-10 


However,  during  the  acquisition  and  operation  phases,  the  model 
must  be  very  representative  (that  is,  very  detailed  in  its 
complexity)  of  the  actual  system  when  the  actual  system  is  not  used 
for  testing.  The  system  can  be  tested  in  an  open  loop  (for 
verifying  that  the  system  can  meet  engineering  specifications)  and 
in  a  closed  loop  (for  verifying  that  the  system  is  effective  in  its 
mission  context).  The  input  data  for  closed-loop  operation  is  much 
more  complex  than  for  open-loop  operation  because  other  systems, 
friendly  and  enemy,  threat,  and  environment,  are  usually  included. 
This  is  especially  true  for  general  war  exercises  at  the  major 
command  level. 

6. 2. 3. 3  Analysis  Utility.  The  objective  of  any  analysis  is  to 
ascertain  quantitative  values  for  the  particular  measures  chosen  to 
evaluate  a  system.  The  values  will  then  be  used  by  the 
decisionmaker  for  decisions  regarding  the  system. 

The  analyst  who  is  charged  with  performing  an  analysis  to 

2 

provide  a  decisionmaker  with  information  concerning  a  particular  C 
mission,  system,  or  subsystem  must  be  aware  of  the  life-cycle  phase 
and  the  type  of  analysis  desired.  Using  Tables  6-1 ,  6-2,  or  6-3, 
the  analyst  can  arrive  at  the  expected  complexity  of  the  analysis. 

These  tables  list  some  sample  measures  to  illustrate  the  focus 
required  for  the  various  categories  of  analysis.  Obviously  no  such 
list  could  be  exhaustive  in  the  sense  of  Identifying  appropriate 
measures  for  every  possible  analysis.  Therefore,  it  is 
advantageous  to  have  a  set  of  guidelines  to  assist  analysts  and 
decisionmakers  in  the  selection  of  MOPs,  MOEs,  and  MOFEs.  Such  a 
set  of  criteria  is  presented  and  discussed  in  the  next  section. 


6-1 1 


6.3  Characteristics  of  Measures 


Performance  and  effectiveness  measures  can  be  characterized  by 
their  physical  and  analytic  attributes. 

Physical  attributes  involve  the  functional  category  to  which 
the  measure  belongs,  its  name,  dimensional  units,  and  numerical 
value.  To  illustrate  the  physical  attributes  of  a  measure,  consider 
a  communications  system  as  an  example.  While  there  is  no 
universally  accepted  procedure  for  functionalizing  communications 
system  measures,  one  frequently  used  method  is  to  subdivide  them 
into  four  categories:  communications  measures,  stability  measures, 
reorganization  measures,  and  security  measures.  An  example  of  a 
specific  measure  within  the  communications  category  might  be  "speed 
of  service,"  whereas  a  useful  stability  measure  might  be  the  "system 
availability."  The  system  "speed  of  service"  could  be  measured  in 
terms  of  its  "expected  time  to  transmit  a  message,"  and  specified  in 
units  of  time. 

Analytic  attributes  are  desirable  characteristics  that  can 
serve  as  a  useful  guide  to  analysts  in  selecting  appropriate 
measures.  The  following  table  (Table  6-^)  provides  a  list  of  such 
desirable  characteristics.  The  first  four  of  the  characteristics 
described  (mission  oriented,  discriminatory,  measurable,  and 
quantitative)  are  particularly  critical  to  successful  analysis,  and 
will  be  addressed  in  further  detail  in  the  subsequent  discussion. 

6.3.1  Key  Characteristics 

6. 3. 1.1  Mission  Oriented.  Effectiveness  measures  are,  by 
definition,  mission  oriented.  The  measure  selected  should  be 
related  to  a  clearly  defined  statement  of  the  mission,  or  objective, 
of  the  system,  or  force,  under  consideration.  This  statement 
provides  explicit  or  implicit  information  regarding  the  standards 
involved.  As  an  example,  let  us  consider  an  analysis  involving  an 


6-12 


TABLE  6-4 


DESIRED  CRITERIA  FOR  MEASURES 


CHARACTERISTICS 

DEFINITION 

Mission  Oriented 

Relate  to  force/system  mission. 

Discriminatory 

Identify  real  differences  between 
alternatives. 

Measurable 

Able  to  be  computed  or  estimated. 

Quantitative 

Able  to  be  assigned  numbers  or  ranked. 

Realistic 

Relate  realistically  to  the  C^  system  and 
associated  uncertainties. 

Objective 

Defined  or  derived,  independent  of 
subjective  opinion.  (It  is  recognized  that 
some  measures  cannot  be  objectively 
defined. ) 

Appropriate 

Relate  to  acceptable  standards  and  analysis 
objectives. 

Sensitive 

Reflect  changes  in  system  variables. 

Inclusive 

Reflect  those  standards  required  by  the 
analysis  objectives. 

Independent 

Mutually  exclusive  with  respect  to  other 
measures. 

Simple 

Easily  understood  by  the  user. 
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acquisition  radar  that  is  part  of  an  air-defense  system.  An 
objective  (or  mission)  of  this  particular  radar  is  to  acquire 
targets  in  the  shortest  possible  time,  given  that  line  of  sight 
exists  between  the  radar  and  the  target.  The  per formance  of  the 
radar  might  be  expressed  in  terms  of  range,  power,  cycle  time, 
discrimination,  and  frequency,  for  example,  whereas  effectiveness  is 
expressed  in  terms  of  the  radar’s  mission.  Under  these 
circumstances,  a  useful  effectiveness  measure  for  a  radar  evaluation 
would  be  the  "time  to  detect  the  target  given  the  existence  of  line 
of  sight."  Of  course,  the  radar  may  have  other  missions,  and  in 
that  case  other  measures  should  be  selected  for  each  of  those  tasks. 

6. 3. 1.2  Discriminatory.  Measures  must  discriminate  sufficiently  so 
that  real  differences  among  alternatives  can  be  readily  identified. 
Without  this  measurement  capability,  important  information  can  be 
obscured.  For  example,  consider  a  comparison  of  two  competing 
air-defense  systems,  e.g.,  one  manual,  the  other  automated.  The 
role  of  the  system  in  the  air-defense  mission  is  to  provide 
direction  to  the  forces.  A  "better"  system  will  result  in 
"better"  air  defense,  in  some  sense  when  a  proper  measure  is  chosen, 
e.g.,  number  of  friendly  fighters  lost  versus  enemy  fighters  shot 
down.  However,  regardless  of  the  control  model  (manual  or 
automated)  employed,  there  might  be  very  little  variation  observed 
in  "number  of  friendly  fighters  lost."  Does  this  mean  that 
automated  control  is  no  better  than  manual  control?  Not 
necessarily.  The  problem  may  very  well  be  that  the  selected  measure 
was  simply  not  sufficiently  sensitive  to  distinguish  between  the  two 
control  systems.  For  that  matter,  unless  friendly  losses  to  enemy 
air  attack  were  suspected  to  be  a  significant  portion  of  to tal 
number  of  friendly  fighters  lost,  major  changes  in  said  losses  due 
to  improving  the  air-defense  system  should  not  be  expected 
a  priori.  An  effectiveness  measure,  however,  that  might  be  more 
discriminating  in  this  case  is  "friendly  losses  to  enemy  air 
attack . " 
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6. 3. 1.3  Measurable.  A  measure  must  represent  a  measurable 
concept.  Data  collection  must  be  possible.  As  a  general  rule, 
"values"  are  assigned  to  measures  on  the  basis  of  observations 
acquired  through  the  use  of  a  broad  range  of  analytic  tools.  These 
include,  but  may  not  be  limited  to,  historical  files,  experiments, 
field-training  exercises,  war  games,  combat,  mathematical  models, 
and  computer  simulations.  The  scenarios  and  combat  models  are 
discussed  in  more  detail  in  the  following  section.  However,  it 
should  be  noted  here  that  certain  combinations  of  scenarios  and 
analytic  tools  may  preclude  (or  certainly  severely  limit)  the 
opportunities  to  acquire  the  data  necessary  to  quantify  a  measure. 

In  a  field-training  exercise,  for  example,  the  inability  to  position 
"observers"  in  a  particular  location  may  deny  the  analyst  access  to 
the  data  needed  to  assign  values  to  a  specific  measure. 

When  computer  simulations  are  used  as  analytic  tools,  certain 
combat  functions  required  to  quantify  measures  may  simply  not  be 
represented  at  all  or  in  adequate  detail.  Finally,  when  conducting 
experimentally  based  evaluations,  the  appropriate  instrumentation 
must  be  available. 

6. 3. 1.1*  Quantitative.  It  is  preferable  for  ease  in  analysis  that 
measures  be  quantifiable .  For  example,  a  numerical  undimensional 
measure  facilitates  both  the  (univariate)  ranking  of  alternatives 
and  the  (multivariate)  combination  of  measures.  The  process  by 
which  the  measures  are  "combined"  is  generally  made  easier  (but 
certainly  not  trivial)  if  the  "values"  of  the  various  measures  can 
be  specified  as  numerical  quantities.  It  is  necessary  that  both  the 
numerical  values  and  the  nature  of  the  relationships  between 
measures  be  specified.  For  example,  how  are  time  and  accuracy 
related?  Is  the  equation  or  set  of  equations  linking  a  set  of 
measures  additive,  multiplicative,  or  of  some  other  functional 
relationships? 
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6.4  Specification  of  Measures 


This  subsection  contains  a  discussion  of  two  approaches  to 
assigning  values  to  measures  of  perforraance/ef fectiveness. 

An  approach  to  the  specification  of  measures  that  focuses  on 
their  hierarchical  relationship  is  illustrated  in  Section  6.5. 

Once  the  type  of  analysis  (mission,  system,  or  subsystem)  has 
been  selected  for  the  life-cycle  phase  of  interest  and  the  desired 
measures  of  performance/effectiveness  have  been  determined,  the 
general  approaches  to  determining  the  values  of  the  measures  for  the 
system  of  interest  are  either  to  use  a  model  of  the  system  of 
interest  with  the  proper  input  data  or,  if  the  system  exists, 
operate  the  actual  system  versus  simulated  data  and  conditions. 

6.4.1  Analysis  Using  System  Models 

System  models  are  either  analytic  models  (closed-form 
mathematical  equations)  or  simulation  models.  The  models  can  vary 
from  a  back-of-the-envelope  description  of  the  physics  of  the 
problem  to  very  large,  complex,  and  detailed  simulations  that 
include  many  and  varied  systems/subsystems,  environmental  effects, 
forces  (both  friendly  and  enemy),  and  procedures.  The  model  should 
produce  values  for  those  variables  required  to  compute  the  selected 
measures,  and  be  constructed  to  the  desired  detail  and  accuracy 
required  for  the  decision  regarding  the  system  being  studied. 

An  illustration  of  a  generalized  system  simulation  model  is 

2 

shown  in  Figure  6-2.  The  input  consists  of  data  for  initial  C 
system  parameters  (system  performance  characteristics,  system 
survivability /endurance  characteristics,  and  strategy  and  tactics) 
and  scenario  data  (environmental  characteristics,  enemy  system 
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FIGURE  6-2 

ANALYSIS  OF  MODELED  C2  SYSTEM 


►  DONE 


6-17 


performance  characteristics,  enemy  system  survivability /endurance 
characteristics,  enemy  strategy  and  tactics,  and  order  of  engagement 
events  versus  time).  The  analysis  process  consists  of  starting  the 
model  with  initial  system  parameter  values  and  scenario  data  and 
computing  the  values  for  the  measures  (possibly  versus  time).  If 
the  computed  measure  values  are  acceptable,  then  the  process 
stops.  If  not,  the  system  parameter  values  are  improved  to 
increase  system  performance  or  effectiveness  and  the  process 
continues  until  the  computed  measure  values  are  as  good  as  the 
desired  measure  values. 

The  model  can  be  used  to  perform  a  tradeoff  analysis  to 
determine  the  most  cost-effective  system  as  described  in 
Figure  6-2.  The  model  can  also  be  used  to  perform  a  sensitivity 
analysis  to  determine  the  effects  of  perturbations  of  system  and 
scenario  parameter  values  on  the  measures  of  performance/ 
effectiveness.  These  sensitivities  will  be  relative  to  the  nominal 
values  assumed  for  the  various  parameters.  (This  is  similar  to 
partial  derivatives  at  a  particular  point  for  a  multivariate 
function.)  The  model  can  also  be  used  to  examine  system  performance 
versus  system  survivability  and  endurance  tradeoffs  for  fixed-system 
cost. 

6.4.2  Analysis  Using  Actual  System 

During  the  acquisition  or  operation  phases,  a  system  analysis 
can  be  performed  on  the  actual  system  being  evaluated.  The  system 
can  be  in  the  R&D  phase,  test  and  evaluation  phase,  or  operation 
phase.  During  the  R&D  phase,  the  decision  on  the  system  being 
developed  is,  "Is  it  feasible  to  continue  R&D  on  this  system?"  In 
the  test  and  evaluation  phase,  the  question  is,  "Is  the  system  ready 
for  operational  deployment?"  In  the  operation  phase,  the  question 
is,  "Is  this  system  satisfying  current  or  planned  operational 
requirements?" 
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6. 4. 2.1  Areas  of  Application.  Testing  can  be  conducted  in  the 
laboratory,  field,  or  after  implementation  in  the  operational 
environment.  The  analysis  process  requires  the  availability  of 
input  data  for  the  system  being  tested.  This  data  can  range  from 
simple  electrical  signals  generated  in  the  laboratory  or  within  the 
system  itself,  such  as  a  stimulator  in  a  radar,  to  very  complex 
scenario  data  generated  for  a  large  force/C^  structure  by  a  large 
computer  simulation. 

For  the  system  analysis  using  the  actual  system,  the  process  is 
essentially  open  loop  in  that  no  iterative  procedures  for  modifying 
the  system  are  used  (see  Figure  6-3).  The  analysis  process 
previously  described  in  this  chapter  should  be  employed  regardless 

p 

of  whether  a  model  of  the  C  system  or  the  actual  system  is  being 
used  in  the  study. 

6 .5  Selection  of  Measures:  Two  Illustrations 

As  previously  mentioned  in  this  paper,  there  are  both 
performance  and  effectiveness  measures.  Effectiveness  measures 
describe  how  well  the  C  system  meets  its  requirements.  Force 
effectiveness  measures  describe  the  contribution  of  the  system  to 
the  overall  force  mission  performance.  Most  military  analysis  is 
force  oriented  in  the  sense  that  the  ultimate  issue  usually  concerns 
the  marginal  contribution  to  force  effectiveness  of  various  systems, 
tactics,  or  force  structure  changes.  Accordingly,  the  analyst's 
goal  is  to  relate  improvement,  for  example,  in  system  performance, 
to  the  ability  of  a  force  to  accomplish  a  specific  mission.  It 
should,  therefore,  not  be  surprising  that  the  focus  of  attention  in 
selecting  measures  for  a  particular  analysis  is  initially  directed 
toward  the  force  and  its  mission. 
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FIGURE  6-3 

ANALYSIS  OF  ACTUAL  C2  SYSTEM 
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6.5.1  Evaluating  a  Tactical  Warfare  C  System 

2 

As  an  example,  let  us  consider  an  air-defense  C  problem. 

Table  6-5  provides  a  hierarchical  structure  of  measures  developed  In 

the  fashion  described  above.  In  this  particular  situation,  assume 

2 

we  are  Interested  In  evaluating  the  contribution  of  the  C  system  to 
force-mission  accomplishment. 

6. 5. 1.1  Level  1 .  The  air-defense  force  consists  of  both  air  and 
ground  elements.  Assuming  that  the  primary  mission  of  air  defense 
Is  to  "protect  friendly  forces,"  two  appropriate  effectiveness 
measures  would  be  "friendly  losses"  or  "friendly  losses  due  to  enemy 
air  attack."  A  measure  sometimes  used  to  assess  the  contribution  of 
friendly  air  defense  Is  "enemy  aircraft  losses."  Such  a  measure, 
however,  falls  to  take  Into  account  that  the  air-defense  force  can 
provide  some  protection  without  necessarily  destroying  enemy 
aircraft.  Specifically,  this  additional  protection  derives  from  the 
"suppression"  capability  of  the  air-defense  force  which  frequently 
reduces  "friendly  losses"  but  does  not  result  In  Increased  "enemy 
aircraft  losses."  While  the  "enemy  aircraft  losses"  Is  potentially 
a  useful  air-defense  system  measure.  If  used  In  Isolation,  It  can 
mask  valuable  Information.  Similar  examples  occur  In  other  areas  of 
military  analysis. 

6. 5. 1.2  Level  2.  Proceeding  down  the  hierarchy,  a  major  component 

of  the  air-defense  force  Is  the  system.  The  system  Is 

responsible  for  allocating  friendly  air-defense  resources  In 

response  to  the  threat  as  It  develops,  while  at  the  same  time 

alerting  the  remainder  of  the  friendly  force  of  Impending  air 

attacks.  Measures  appropriate  for  the  former  task  are:  "fraction 

of  engagement  opportunities  exploited,"  and  "fraction  of  air-defense 

resources  needed  to  respond  to  a  given  level  of  threat."  An  example 

of  a  measure  that  addresses  the  alerting  mission  of  the  air-defense 
2 

C  system  Is  the  "early  warning  time"  afforded  the  friendly  force. 
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TABLE  6-5 


MEASURE  HIERARCHY 


HIERARCHY 

LEVEL 

SYSTEM  LEVEL 

MISSION/TASK 

MEASURES 

1 

Air  Defense 
Force 

Protect  Friendly 
Forces 

Friendly  Losses 

2 

C^  System 

a.  Alert  Forces 

b.  Resource 
Allocation 

A.  Early  Warning  Time 
b.  Fraction  of  Engage¬ 
ment  Opportunities 
Exploited 

3 

Acquisition 

Radar 

Detection 

Time  to  Detect 
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6.5.1  .3  Level  3 .  At  the  bottom  of  the  hierarchy  is  the  acquisition 

2 

radar,  an  element  of  the  air-defense  C  system.  As  we  have 
previously  seen,  the  primary  task  of  the  radar  is  to  detect  targets 
in  the  shortest  possible  time.  An  appropriate  measure  is  the  "time 
to  detect,  given  line  of  sight  to  the  target." 

p 

6.5.2  Evaluating  a  Strategic  C  System 

The  selection  of  effectiveness  measures  for  strategic  systems 
is  usually  more  straightforward  than  for  tactical  systems,  primarily 
because  the  strategic  mission  is  traditionally  open  ended  with 
little  feedback.  Relating  this  to  our  conceptual  model,  we  are 
often  not  interested  in  feedback  into  the  environment  with  further 
sensing  changing  the  decision.  This  view  may  be  simplistic  but,  for 
Positive  Control  Launch  for  survival  of  strategic  systems,  it  is 
usually  true.  For  this  example,  assume  an  interest  in  evaluating 
the  contribution  of  a  new  Tactical  Warning /At tack  Assessment  (TW/AA) 
communications  system  to  force-mission  accomplishment.  The  starting 
hypothesis  is  that  warning  data  from  sensor  sites  is  not  reaching 
the  decisionmakers  in  sufficient  time  to  ensure  mission 
accomplishment.  Since  the  major  time  delay  is  in  the  communications 
subsystem  in  a  Jamming  and  nuclear  environment,  the  focus  is  on  that 
element  of  the  force.  Table  6-6  describes  a  hierarchy  of  measures 
for  such  a  situation. 

6. 5. 2.1  Level  1 .  To  define  measures,  the  mission  of  the  force  must 
be  clearly  defined.  A  broad  statement  of  the  strategic  mission  such 
as  "deter  aggression"  lacks  sufficient  detail  to  use  as  a  measure  of 
effectiveness  or  performance.  Breaking  the  mission  into  "ensuring 
enough  forces  survive,  and  executing  war  orders  to  inflict 
unacceptable  damage  on  an  aggressor"  refines  the  mission  but  does 
not  provide  an  easily  measurable  yardstick  of  system  performance, 
i.e.,  what  is  unacceptable.  More  practical  operational  measures 
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TABLE  6-6 


MEASURES  FOR  STRATEGIC  SYSTEMS 


HIERARCHY 

LEVEL 

SYSTEM  LEVEL 

MISSION/TASK 

MEASURES 

1 

Strategic 

Force 

a.  Deterrence 

b.  Provide  for 

Surv.  of  Force 

c.  Execute  the  War 
Order 

a.  Damage  to  an 

Aggressor  Compared 
to  Gains 

b.  Weapons  Surviving 
and  Connected 

c.  Weapons  on  Target 

2 

C^  System 
(TW/AA  and 
Command 
Centers) 

a.  Track  All  Threats 

b.  Alert  Decision 

a.  Sensor  Coverage 

b.  Warning  Time 

c.  Data  Correctness 

d.  Decision  Time 

3 

Comm.  Element 

a.  Connect  Sensor  to 
TW/AA  System 

a.  Time  to  Transmit 

b.  Reliability 
Availability,  etc. 

c.  Manpower 

a.  Additional  Weapons 
Surviving  Based  on 
Comm.  System 
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would  be  "%  or  number  of  weapons  surviving"  or  or  number  of 
weapons  on  target."  To  reflect  the  importance  of  communications  we 
can  further  refine  the  force  measure  to  weapons  surviving  and 
connected.  These  measures  can  be  combined  with  other  weapons  system 
and  personnel  measures  to  define  an  overall  force  measure  of  damage 
to  an  aggressor  compared  to  his  gains.  Using  the  definitions  from 
Chapter  2,  these  force-level  measures  are  MOFEs. 

p 

6. 5. 2. 2  Level  2.  A  major  piece  (subsystem)  of  this  force  is  the  C" 
system.  Our  interest  is  in  the  TW/AA  subsystem  whose  mission  is  to 
identify  and  track  threats,  alert  the  forces,  and  provide 
recommendations  to  decisionmakers.  Appropriate  measures  would  be 
response  time,  percentage  of  threats  ttie  system  can  handle,  and 
saturation.  Examples  of  goals  which  can  be  measured  include  receipt 
of  data,  processing  and  forwarding  to  decisionmakers  in  five 
minutes;  tracking  Intercontinental  Ballistic  Missiles,  Sea-Launched 
Ballistic  Missiles  (SLBMs) ,  bombers  and  cruise  missiles  over  one 
square  meter  radar  cross  section  and  above  a  300-foot  altitude;  or 
handling  1  ,000  targets  simultaneously.  A  less  obvious  measure 
stemming  from  the  force  mission  of  deterrence  would  be  false-alarm 
detection  rate.  Launching  a  counterattack  against  a  non-existent 
aggressor  starts  war,  it  does  not  deter  it. 

6. 5. 2. 3  Level  3 .  The  focal  point  of  this  analysis  is  the 
contributions  of  a  new  communications  system  to  TW/AA  and  force 
effectiveness.  This  is  the  lowest  level  of  the  analysis,  vhich  will 
be  called  the  subsystem  level.  Requiring  the  component  to  send  a 
message  from  point  A  to  point  B  is  not  sufficient.  The  focus  is  now 
directed  toward  reliability,  maintainability,  timeliness,  usability, 
and  other  aspects  of  system  specifications.  At  the  lowest  level, 
the  most  obvious  classes  of  measures  address  the  following 
questions: 
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a.  How  well  does  the  equipment  performance  (physical 
parameters)  meet  system  performance  requirements? 

b.  Can  the  system  be  operated  by  the  two  men  per  transmitter 
available  to  the  user? 

c.  Will  this  system  overload  the  processing  center? 

d.  Does  the  system  provide  99?  reliability,  98?  availability, 
and  90?  accuracy  of  transmitting  data  from  a  particular 
sensor  to  the  processor? 

e.  Will  the  system  create  less  than  1  in  100  million  wrong 
messages  in  jamming  and  nuclear  environments? 

Measures  of  75  baud  minimum  data  rate,  six  messages 
simultaneously  (traffic  volume),  and  availability  of  98?  at  the 
subsystem  level  can  be  directly  related  to  MOFEs.  Weapons 
surviving,  especially  aircraft  resources  escaping  their  bases  under 
an  SLBM  and  cruise  missile  attack,  can  be  directly  related  to  how 
fast  the  communications  system  sends  warning  data  from  the  sensors 
to  the  processing  centers  and  then  to  the  decisionmakers.  Human 
decision  time  measures  at  the  force  level  may,  at  this  point,  render 
this  analysis  useless  if  decision  time  rather  than  the  communications 
system  starts  to  drive  the  scenario. 
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7.0  INTRODUCTION 

Beyond  developing  a  conceptual  framework  for  viewing  a  C^ 

process,  it  is  important  to  consider  a  mathematical/heuristic  method 

that  can  be  used  for  design,  evaluation,  or  tradeoff  studies.  For 
2 

this  purpose,  a  C  system  can  be  represented  abstractly  as  a 
transformation  that  maps  a  collection  of  inputs  (primarily  in  the 
form  of  sensed  data  and  status  reports)  into  outputs  (generally  in 
the  form  of  plans  and  schedules  for  effecting  resource  alloca¬ 
tions).  Another  set  of  inputs,  referred  to  here  as  internals, 
includes  mission  definition,  objectives  and  characteristics, 
historical  data  bases,  standards,  and  procedures  that  will  be  viewed 
as  resident  in  the  system  and  fixed  during  the  period  of  interest. 

As  stated  above,  this  methodology  is  Intended  for  use  in 
design,  evaluation,  and  tradeoff  analyses; 

a.  The  design  problem  begins  by  defining  desired  outputs  and 
measures  which  will  be  used  to  assess  system  performance 
against  design  objectives.  The  inputs  and  internals  needed 
to  provide  the  data  and  information  required  to  produce 
those  outputs  and  the  mechanism  for  transforming  the  inputs 
into  the  desired  outputs  are  then  derived.  The  initial 
design  approach  is  often  top-down,  with  the  designer  using 
organizational,,  functional,  spatial,  or  temporal 
decomposition  to  identify  the  required  inputs,  internals, 
and  transformations. 
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b.  Operators  use  systems  to  perform  a  mission.  In  operations 
analysis,  perfomance  of  the  system  is  often  evaluated 
using  a  bottom-up  approach  wherein  low-level  performance 
measures  are  combined  to  determine  system  performance. 
Designers  also  use  this  approach  in  detail  design. 

c.  In  the  acquisition  stage  of  the  system  life  cycle,  both 
top-down  and  bottom-up  approaches  to  analysis  are  needed. 

The  top-down  approach  assists  the  program  manager  to  focus 
on  those  system  elements  which  most  affect  overall 
effectiveness  when  making  cost-effectiveness 
calculations.  The  botton-up  approach  allows  comparative 
evaluation  of  alternative  well-defined  competing 
approaches. 

7.1  MOEs  -  Attributes  and  Requirements 

In  the  previous  sections,  a  conceptual  framework  for  the 
development  and  use  of  models  for  generating  quantities  to  be  used 
in  measuring  effectiveness  was  presented.  Specifically,  it  was 
shown  that  a  mission  has  implicit  in  its  definition  the  attainment 
of  one  or  more  goals.  The  achievement  of  these  goals,  in  turn, 
gives  rise  to  a  set  of  critical  attributes  that  the  system  must 
possess.  These  attributes  can  be  very  general,  as  concepts,  but 
their  quantitative  interpretation  is  invariably  highly  mission 
dependent.  For  example,  the  attributes  of  timeliness, 
responsiveness,  or  robustness  convey  intuitive  concepts  rather 
well.  However,  as  Lawson  (1981)  has  written,  "in  a  typical 
discussion  of  C^,  it  is  taken  as  axiomatic  that  the  information 
presented  to  the  commander  must  be  timely  as  well  as  accurate, 
complete,  etc.  ....  Little  or  nothing  is  said  about  how  timely 
is  timely  enough  .  .  .  ."  Thus  we  need  to  go  a  step  further  and 
define  a  set  of  variables  that  expresses  concretely  these  concepts 
for  a  given  mission  context.  For  example,  the  ratio  of  the  time  to 
cycle  through  the  process  for  a  particular  task  to  the  inter¬ 
arrival  time  of  tasks  may  be  one  of  the  variables  that  is  key  to  the 
concept  of  timeliness  for  a  given  mission.  Moreover,  it  may  occur 
that  more  than  one  variable  is  needed  to  interpret  a  given  attribute. 
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Often,  it  is  also  necessary  to  consider  tradeoffs  between 
attributes  such  as  timeliness  and  completeness  or  timeliness  and 
accuracy.  Again  this  will  depend  on  definitions  of  variables  that 
interpret  more  precisely  the  attributes  for  a  given  mission  to  allow 

p 

comparisons  among  values  resulting  from  different  C  system 
realizations.  These  variables  will  be  called  "Variables  for 
Measuring  Effectiveness"  (VFMEs). 

The  basic  approach  to  developing  attributes  and  converting  them 
into  requirements  for  design  or  evaluation  appears  to  consist  of 
three  steps: 

a.  Extracting  from  an  expert  (e.g.,  the  responsible  commander) 
the  attributes  of  that  are  essential  to  achieve  desired 
mission  performance. 

p 

b.  Associating  with  each  of  these  C  attributes  specific 
variables  that  measure  the  performance  of  the  attribute. 

c.  Using  simulation  to  establish  acceptable  values  for  the 
variables  to  achieve  required  mission  performance. 

Typically  this  step  will  result  in  intervals  for  each 
variable,  such  as  probability  of  detection  greater  than 
some  value. 

If  not  carefully  treated,  this  last  step  can  lead  to  errors  in 
system  design.  These  errors  arise  because  the  intervals  chosen  to 
establish  requirements  do  not,  in  general,  account  for  the  depen¬ 
dence  among  the  variables.  Thus,  as  portrayed  in  Figure  7“1 , 

1  2 

v^  and  v^  both  acceptable  values  for  the  variable  v^  but  for 

different,  thougi  overlapping,  values  of  ^2’  Thus,  if  one  merely 

projects  the  acceptable  region  onto  the  v^  and  V2  axes  to  get 

acceptable  intervals  to  define  requirements  of  the  variables  v-j  and 

V2,  the  system  designed  may  actually  give  rise  to  a 
*  * 

pair  (v^ ,  V  )  that  lies  outside  the  acceptable  region.  It  is 
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FIGURE  7-1 

VARIABLES  OF  MISSION  PERFORMANCE 
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believed  that  this  is  the  cause  of  failures  in  many  system  designs 
which  integrate  a  collection  of  subsystems,  each  of  which  seems  to 
be  properly  designed  and  meets  the  design  specifications.  This 
issue  will  be  addressed  again  later  in  this  chapter. 

7 . 2  A  Probabilistic  Formulation 

This  section  presents  a  mathematical  theory  to  determine 
(1)  when  sufficient  attributes  and  variables  for  measuring  them  have 
been  defined;  (2)  how  to  use  these  variables  to  set  requirements, 

(3)  how  to  define  mathematically  a  system  measure  of  effectiveness, 
and  (^)  how  to  use  the  definition  to  evaluate  system  designs. 

7.2.1  Sufficiency  for  the  Set  of  Variables  for  Measuring 
Effectiveness 

Assume  that  the  combat  situation  of  interest  has  been  modeled 

p 

in  detail  in  such  a  way  that  the  various  C  processes  and  systems 
can  be  included  in  a  realistic  fashion.  Call  this  model  M. 
Conceptually,  this  model  is  viewed  as  a  monte-carlo  simulation  which 
gives  a  set  of  combat  outcomes  for  each  replication. 

Let  S  be  the  set  of  all  C^  system  realizations  of  interest, 
including  all  variations  of  the  same  system  obtained  by  taking 
variations  of  the  operating  parameters. 

The  set  {m,S},  consisting  of  the  simulation  M  and  a  well- 
2 

defined  set,  S,  of  C  systems,  which  will  serve  as  the  universe  of 
system  alternatives,  provides  a  well-defined  context  for  analysis 
purposes . 

Let  ^  =  (v^ ,  ...  ,  Vjjj)  be  a  vector  of  the  given  VFMEs  of 
command  and  control  that  express  the  important  mission-dependent 
notions  of  completeness,  accuracy,  timeliness,  and  so  on.  The 
vector  ^  is  considered  to  be  random,  since  each  replication  will 
provide  a  simulated  real-world  outcome  with  different  time  delays, 
targets  detected,  etc.  Let  £  =  (c-] ,  ...  ,  c^^)  be  the  random  vector 
of  combat  outcomes  of  interest  to  the  operational  commander  (e.g., 
enemy  resources  destroyed,  own  resources  lost). 
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For  each  replication  of  the  simulation  M  with  C  system  s  G  S, 
one  obtains  realizations,  i.e.,  values,  for  _v  and  g_.  By  taking 
enough  replications  one  can  estimate  the  joint  distribution  function 
for  V  and  c  and,  more  important,  the  conditional  probability 
measure,  u  for  c  given  v  and  s.  This  means  that  for  a  set,  0,  of 
combat  outcomes  one  can  derive  the  probability, 

u(0lv,s), 

that  an  outcome,  £  G  0,  will  occur,  given  v  and  s. 

Definition:  the  VFMEs,  which  are  the  elements  of  _v,  are  said  to  be 
sufficient  for  measuring  effectiveness  if  ii(-  |v,  s)  does  not  depend 
upon  s  f or  s  G  S,  i.e.,  if  p( • | v,  s)  =  y( • [ v) . 

Intuitively,  this  means  that  if  v  is  sufficient  for  measuring 
effectiveness,  the  essence  of  the  system  within  the  universe  has 
been  captured  by  v.  Clearly  one  seeks  the  sufficient  vector  _v  with 
lowest  dimension. 

7.2.2  Setting  Requirements 

Next,  it  is  necessary  to  establish  requirements  for  values  of 
the  VFMEs  so  that  individual  components  of  the  system  may  be 
designed  with  the  assurance  that,  when  the  total  system  is 
constructed,  it  will  achieve  the  desired  objectives. 

The  requirements  are  set  as  follows.  The  "military  expert" 
establishes  a  set  of  desired  combat  objectives  to  be  achieved,  with 
some  probability,  in  terms  of  the  vector  £.  For  example,  let  c^  be 
the  percentage  of  planned  targets  destroyed,  and  C2  be  the  percen¬ 
tage  of  own  aircraft  lost.  Then  the  objective  set  might 
be  0  =  (c^  ,  c^),  where 

c^  >  30%,  and 

^  10?, 

with  probability  of  being  in  0,0.95  or  higher. 
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The  conditional  probability  measure,  p,  which  is  independent  of 

p 

any  particular  C  system  s  e  S  because  ^  is  sufficient,  is  used  to 
define  requirements.  The  requirements  set,  R,  is  defined  by 

R  =  {v  :  u(0|v)  >  .95} 

All  tradeoffs  in  attributes  are  implicit  in  R. 

2 

7.2.3  Measuring  C  System  Effectiveness 

2 

Returning  to  a  specific  C  system  s  e  S,  it  is  desired  to 
define  its  measure  of  effectiveness  so  that  systems  can  be  rank 
ordered.  For  a  set  of  objectives  and  a  particular  vector  of  VFMEs 
V,  it  has  been  shown  how  to  define  the  probability  of  achieving  the 
objectives,  p(0|^),  and  how  to  define  the  requirement  set  R. 

Effectiveness,  E(s),  of  s  G  S  is  now  defined  to  be 

E(s)  =  Iy)c1F3(y) 

2 

where  Fg  is  the  probability  distribution  of  the  VFMEs  for  the  C 

p 

system  s.  Note  that  a  C  system  that  causes  ^  to  fall  outside  R 

2 

with  high  probability  will  receive  a  low  value  for  E  and  a  C  system 
that  meets  the  requirements  with  high  probability  will  receive  a 
high  value,  which  is  consistent  with  what  a  measure  of  effectiveness 
should  provide. 

7.2.^  Evaluating  System  Designs 

This  measure  of  effectiveness,  E,  allows  a  comparison  between 
potential  systems  based  on  how  they  influence  combat  outcomes. 
Moreover,  if  S  consists  of  a  finite  number  of  candidate  systems,  an 
"optimal"  choice  can  be  made. 

7 . 3  A  Constructive  Approach 

The  last  section  presented  a  mathematical  formulation  that 
allows  the  definition  of  a  requirements  set  and  a  measure  of 
effectiveness  for  use  in  comparing  system  alternatives.  This 
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section  presents  a  constructive  approach  for  making  such 
comparisons. 

Let  the  VFMEs  of  the  system  be  denoted  by  the  vector  _v  (the 
vector  V  is  assumed  to  be  sufficient  in  the  sense  of  Section 
7.3.1).  For  simplicity  of  presentation,  it  will  be  assumed  that  the 
variables  take  values  that  are  real  numbers  (^eRjjj).  However,  this 
need  not  be  the  case;  some  of  them  may  take  linguistic  values  such 
as  "fast"  or  "slow."  Let  the  environment  be  described  by  a 
vector  r^. 

7.3.1  Setting  Requirements 

Requirements  can  be  obtained  by  setting  values  (or  ranges  of 
values)  for  the  variables  v^^.  In  order  to  do  that,  however,  it  is 
necessary  to  go  outside  the  boundary  of  the  C  process.  A  set  of 
models  is  required  that  allows  the  mapping  of  the  variables  to 
the  variables  for  measuring  mission  outcomes,  The  values  of 

these  variables  are  determined  from  the  combat  models  as  the 
variables  ^  and  the  environment  descriptors  £  are  varied,  that  is: 

V,  r  c 

This  mapping  can  be  represented  pictorially  as  shown  in  Figure  7~2. 

If  the  vector  £  can  take  values,  v  G  V  C  R^,  then  the 
variables  c^  can  take  values  over  a  corresponding  range  in  their 
space,  i.e.,  given  £  for 

vGVCR,  cgCCR 

m  -  n 

Graphically,  if  the  vector,  £,  is  two  dimensional  and  the  vector,  £, 
is  three  dimensional,  then  the  region  V  in  the  left  side  (a)  of 
Figure  7-3  maps  onto  the  region  C  in  the  right  side  (b)  of 
Figure  7-3- 

For  example,  consider  a  case  with  two  such  measures:  c^ ,  that 
reflects  the  degree  of  success  in  destroying  targets,  and  C2,  that 
reflects  survivability  through  a  measure  of  own  aircraft  lost. 
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Then,  for  a  given  subset  in  the  space  defined  by  the  vector,  the 
evaluation  of  Ci  and  C2  can  be  obtained  through  the  use  of  battle 
simulation  models.  These  models  should  be  as  complete  as  is 
required  for  the  problem  at  hand.  Therefore,  for  example,  they  may 
have  embedded  in  them  models  of  the  Red  force. 

The  locus  in  the  space  of  the  mission  measures  shows  the 

tradeoff  between  success  and  survivability.  Assume  the  command  has 

* 

established  an  acceptable  level  of  losses  (c„  ^  c^)  and  a  minimum 
degree  of  success  (c^  s  c^). 

The  cross-hatched  regions  in  Figure  7-4  (a,c)  indicate  the 
unacceptable  portions  of  the  tradeoff  region.  In  this  example, 
there  is  an  acceptable  range  where  satisfactory  performance  in 
meeting  mission  objectives  can  be  achieved  at  an  acceptable  level  of 
losses.  This  range  can  be  realized,  if  the  vector  ^  takes  values  in 
the  portion  of  the  region  V  that  is  not  cross-hatched. 

While  many  different  models  may  be  used  to  obtain  values  for 
the  quantities  c^,  it  is  important  that  consistent  sets  of  values  be 
obtained.  This  implies  that  the  various  models  should  be  exercised 
for  the  same  values  of  jr  and  r^  to  obtain  a  mutually  consistent  set 
of  values  for  the  variables  Cj.  Canparison  between  systems  can  be 
made  by  varying  the  variables  for  a  fixed  mission  and  fixed  set 
of  environmental  conditions. 

If  the  environmental  parameters  £  are  changed  and  the  procedure 
repeated,  then  a  new  region,  C,  will  be  obtained.  Indeed,  a  whole 
family  of  regions  can  be  obtained  as  the  environmental  parameters 
vary.  Each  region  characterizes  the  performance  capabilities  of  the 
process  for  a  given  mission  and  for  a  given  set  of  envirormental 
conditions  (or  context). 

Thus,  one  can  derive  a  family  of  regions,  V,  parameterized  over 
the  environment  vector  £.  Each  such  region  yields  a  tradeoff 
locus  C.  In  general,  some  of  the  loci  will  have  no  acceptable 
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range  (Figure  7-5(a)),  while  for  other  values  of  £  the  whole  locus 
may  be  acceptable,  as  shown  in  Figure  7-5 (b). 

Extreme  cases  for  the  illustrative  example  would  be: 

a.  Weather  conditions  and  enemy  defenses  are  such  that  the 
expected  value  of  targets  destroyed  is  very  low,  while 
losses  from  enemy  defenses  and/or  weather  conditions  are 
exceedingly  higi  (Figure  7-5(a)). 

b.  Weather  conditions  are  excellent  and  the  opposing  forces 
have  no  anti-air  capability  (Figure  7-5(b)). 

Considerations  of  the  admissible  portions  of  the  tradeoff 
locus,  C,  parameterized  over  the  vector  £,  lead  to  the  admissible 
ranges  of  the  vector  _v  G  V.  Since,  in  general,  it  may  not  be 
possible  to  invert  the  mapping  from  v_  to  £,  the  correspondence 
between  them  is  obtained  when  the  mapping  from  £  to  £  is  obtained 
through  means  such  as  simulations,  exercises,  and  war  games. 

The  set  of  admissible  values, Va,  of  the  variables  for  measuring 
effectiveness,  of  the  process  leads  directly  to  requirements  of 
the  general  form: 

V  <  V.  <  V. , 

lO  -1  i1 

2 

With  the  setting  of  requirements  for  the  C  process,  it  is  now 
possible  to  address  the  question  of  effectiveness. 

7.3.2  Measures  of  Effectiveness 

Implicit  in  the  notion  of  assessment  or  evaluation  or  measuring 

p 

the  effectiveness  of  C  system  is  the  concept  of  a  standard.  If  the 
requirements  represent  a  standard,  then  comparison  of  the  measures 
of  performance  to  the  corresponding  requirements  for  these  measures 
leads  to  measures  of  effectiveness.  Scxnetimes  the  comparison  is 
explicit,  when  one  measures  by  how  much  a  measure  of  performance 
exceeds  a  given  level  of  performance.  Scanetimes  it  is  implicit, 
when  the  measure  itself  is  a  deviation  such  as  the  probability  of 
error . 
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FIGURE  7-5 


Up  to  now,  a  top-down  approach  has  been  used:  the  mission  goal 

p 

was  expressed  in  terras  of  several  objectives,  the  C  process  was 
characterized  by  a  number  of  attributes,  these  attributes  were 
expressed  in  terns  of  measurable  variables  and,  finally, 
requirements  were  established,  i.e.,  admissible  ranges  for  the 
values  of  these  variables.  The  system  developers  then  design 

p 

systems  that  support  the  functions  of  the  C  process. 

It  has  already  been  shown  in  the  previous  subsection  how  the 
requirements  can  be  defined  from  a  locus,  i.e.,  a  region,  V_,  in  the 

Oi 

space  defined  by  the  vector  In  an  analogous  manner,  there  is  a 
locus,  Vg,  in  that  same  space  that  is  defined  by  the  values  of  _v 
that  can  be  realized  by  alternative  system  designs. 

2 

To  obtain  Vg  let  the  parameters  of  the  C  system,  as  defined  in 
Chapter  2,  be  denoted  by  the  vector,  £.  Again,  for  simplicity  of 
presentation,  it  is  assumed  that  the  parameters  take  values  that  are 
real  numbers,  i.e.,  £  ^R|<*  Using  simulations  that  relate  system 
performance  as  measured  by  the  variables,  to  the  system 
parameters,  £,  one  can  determine  the  values  of  _v  as  the  parameters, 
£,  and  the  environmental  descriptors,  £,  are  varied  as  indicated 
below  and  in  Figure  7-6. 

E,  r  ^  V. 

In  this  way  one  can  determine  the  set,  V„,  which  includes  the 

D 

values  for  the  VFMEs  that  are  achieved  for  a  class  of  designs. 
Summarizing,  let  the  set  of  admissible  values  (for  acceptable 
mission  outcomes)  of  ^  be  denoted  by  and  let  the  set  of  values  of 

V  realized  by  a  system  design  be  denoted  by  V„.  In  general, 

V  ^  V  ,  and  acceptable  system  designs  will  only  correspond  to  those 

S  3 

which  have  parameters,  £,  such  that 

E  V  e  V  n  V  . 

-  s  a 
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The  necessary  ingredients  are  now  in  hand  to  consider  the 
problem  of  evaluating  overall  system  effectiveness  in  the  presence 
of  multiple  criteria.  One  way  that  a  comparison  can  be  made  is  by 
analyzing  separately  each  dimension,  i.e.,  each  of  the  VFMEs.  A 
metric  can  be  established  for  each  dimension  and  a  value 

p 

calculated.  For  example,  the  C  system  may  exceed  substantially  the 
timeliness  requirements,  may  barely  satisfy  the  robustness  and 
flexibility  requirements,  may  be  responsive,  but  can  only  support 
one  mission  at  a  time.  How  does  one  establish  an  absolute  measure 
of  effectiveness,  i.e.,  this  system  is  or  is  not  very  effective,  and 
how  does  one  compare  two  alternative  systems,  i.e.,  this  syston  is 
more  effective  than  that? 

The  existence  of  a  vector  of  VFMEs  leads  to  both  conceptual  and 
technical  problems  in  evaluating  systems.  There  are  problems 
associated  with  attempts  to  map  the  vector  into  a  scalar  by 
considering  weighted  averages  of  the  components  of  the  vector. 

There  are  issues  associated  with  trading  off  between  variables 
without  due  consideration  to  their  scaling.  In  addition  to  these 
issues,  which  arise  in  many  of  the  commonly  used  approaches,  there 
are  subtle  issues  related  to  the  fact  that,  while  each  component  of 
the  vector  y  may  take  values  over  a  range,  the  n-tuple  itself  that 
corresponds  to  the  vector  is  constrained  to  lie  on  a  surface  or 
locus.  This  means  that  one  cannot  consider  each  variable  as  being 
able  to  take  values  anywhere  in  its  range,  independently  of  the 
values  taken  by  the  other  variables  (see  Figure  7-1). 

One  possible  approach  that  avoids  these  problems  is  based  on  an 
intuitive  notion.  If  a  system  meets  or  exceeds  all  the  performance 
requirements  derived  from  the  considerations  described  in 
Section  7.3.1,  then  this  would  be  an  effective  system.  If  a  system 
does  not  meet  any  of  the  requirements,  then  it  is  ineffective. 

Since  a  system’s  performance  is  not  characterized  by  a  single  point 
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in  the  space  defined  by  the  VFMEs,  the  usual  case  is  that  one 
portion  of  the  locus  Vg ,  the  performance  locus,  will  satisfy  all 
requirements,  while  the  other  portion  will  satisfy  only  some  of  the 
requirements.  A  possible  means  for  ordering  alternative  systems  and 
for  computing  an  absolute  value  of  effectiveness  is  to  measure  the 
extent  to  which  the  locus  Vg  lies  within  the  admissible  region  V^. 

For  example,  as  one  varies  the  environmental  parameters  r_,  one 
obtains  different  values  of  If  most  of  these  values  of  v  meet 
the  requirements,  then  the  system  would  be  effective  for  this 

mission.  Furthermore,  one  may  parameterize  over  missions  and  again 
compare  the  resulting  set  of  v  values  to  the  requirements. 

Mathematically,  we  define  a  function,  m  with  domain  V.  This 
function  may  be,  for  example,  the  area  of  the  surface  defined  by  Vg, 
or,  if  Vg  consists  of  a  finite  set  of  points,  it  may  be  the  number 
of  points  in  Vg. 

Now  consider  the  portion  of  Vg,  denoted  by  Vg,  that  meets  the 
requirements,  i.e.,  the  portion  of  the  surface  that  is  within  the 
region  defined  by  the  requirements.  This  can  be  expressed  as  the 
intersection  of  the  two  sets  (or  loci) 


V  =  V  n  V 
e  s  a 


If  all  points  satisfy  the  requirements,  then 


Ve  =  Vs 


If  no  points  satisfy  the  requirements,  then 


V  =  d) 
e 


A  scaled  measure  of  effectiveness  is  then  the  fraction  of  the  system 
performance  locus  that  satisfies  the  requirements: 


MOE  = 


m(V  )  m(V  n  V  ) 
e  s  a 


m  ( V  )  ni  ( V  ) 
s  s 
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This  very  simple  measure  does  not  distinguish  between  values  of 
_v  that  barely  exceed  the  requirements  and  values  of  _v  that 
significantly  exceed  the  requirements.  These  considerations  could 
be  modeled  through  a  weighting  function  w(v;) ,  introduced  into  the 
function  m,  that  assigns  different  weights  to  different  portions  of 
the  requirements  locus.  This  approach  also  would  allow,  for 
example,  accommodation  of  styles  of  command,  expressed  in  terms  of 
risk-taking  behavior,  such  as  intuitive  versus  deliberate  styles,  if 
they  are  known  or  elicited  from  the  commander. 

7.3.3  Sensitivity  Analysis 

The  mathematical  framework  described  herein  allows  the 
evaluation  of  the  sensitivity  of  the  above-defined  MOE  to  the 
variables  for  measuring  effectiveness.  This  can  be  expressed 
formally  as  the  ability  to  take  the  derivative  of  the  MOE  with 
respect  to  the  vector 

d  MOE 
dv 

Of  course,  this  is  only  a  formal  expression  since  the  MOE  may 
not  be  differentiable.  Differences  would  then  be  used: 

AMOE 

Av 

A  more  Interesting  result,  that  allows  one  to  confirm 
experience  from  the  user  and  the  systan  developer  communities,  is 
the  calculation  of  the  sensitivity  of  the  MOEs  to  system 
parameters: 

d  MOE  _  3M0E  ^  ^ 
d£  “  9v  9£ 

The  last  term  on  the  ri^t  expresses  changes  in  the  variables 
for  measuring  effectiveness  to  changes  in  the  system  parameters, 
i.e.,  it  reflects  the  sensitivity  of  these  variables  to  the  system 
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parameters.  These  are  quantities  that  the  system  developers  usually 
estimate  or  calculate  as  part  of  the  design  process.  The  preceding 
term  is  the  quantity  that  analysts  would  calculate. 

7.4  Conclusion 

In  this  section,  a  mathematical  framework  has  been  outlined 
that  attempts  to  interpret  some  of  the  technical  issues  in  measuring 
effectiveness.  The  mathematical  formalisms  (probability,  vectors, 
sets,  spaces,  surfaces)  were  chosen  only  for  illustrative  purposes 
and  in  order  to  make  the  discussion  more  concrete.  However,  there 
is  no  requirement  that  the  variables  must  take  real,  numerical 
values,  that  they  be  continuous,  or  that  the  various  mappings  be 
constrained  to  be  functions  of  real  variables.  Indeed,  other 
formalisms  have  been  proposed  and  can  be  used,  e.g.,  fuzzy  sets,  as 
appropriate. 
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CHAPTER  8 


SUMMARY 

by 

Dr.  Michael  G.  Sovereign 
Dr.  Ricki  Sweet 

The  hypothesis  contained  in  the  previous  chapters  is  that  there  is 

2 

a  generic  structure  which  has  great  utility  in  the  evaluation  of  C 
systems.  This  chapter  briefly  reviews  this  structure  and  the  status  of 
its  proposed  modular  building  blocks.  We  will  then  propose  that  a  test 
of  this  hypothesis  be  conducted  in  another  Workshop  in  January  1986. 

The  architecture  of  this  report,  as  discussed  in  Chapter  3  and 

again  shown  as  Figure  8-1 ,  is  based  on  a  suggested  set  of  definitions 

for  C^  system  evaluation  in  Chapter  2.  These  definitions  provide  a 

means  of  specifying  both  the  system  under  evaluation  and  the  measures 

for  evaluation.  They  are  a  function  of  vftiere  the  boundary  is  drawn 

around  the  system  at  hand.  This  approach  is  common  to  all  disciplines 

which  use  systems  concepts.  The  bulk  of  this  report  describes  the  use 

2 

of  these  concepts  to  better  evaluate  C  systems. 

The  evaluation  structure  is  application  driven.  We  believe  that 

such  a  structure  is  most  needed  in  the  difficult  task  of  preparing 

comparable  measures  about  dissimilar  systems  vrfiich  are  competing 

2 

alternatives  for  budget  decisions,  i.e.,  C  versus  weapons  systems. 

A  specific  application  and  distinction  of  systems  boundaries 
provides  a  basis  for  specifying  these  analytical  modules  for  the 
important  development  and  generation  of  MOEs.  Data  sources,  parameter 
types,  and  mathematical  formulations  then  follow.  The  ultimate  goal 
will  be  to  identify  the  mix  and  match  of  applications,  boundary 
conditions,  models,  and  measures  as  well  as  techniques  for  data 
collection.  Such  a  "menu"  approach  facilitates  the  structuring  of 
analyses  and  provides  clearer  explication  and  agreement  on  the  analytic 
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What  is  the  Structure? 


(0 
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FIGURE  8-1 

STRUCTURE  —  ARCHITECTURE 


approach  being  used.  This,  in  turn,  properly  focuses  attention  upon 
assumptions  being  made,  analytical  factors,  and  system  parameters,  and 
the  results  obtained.  (See  Figure  8-6.) 

The  evaluation  structure  remains  incomplete.  Additionally,  the 
approach  generated  needs  to  be  applied  to  some  real-world  problems  to 
assess  its  utility  and  generality.  The  results  of  the  applications 
should  be  assessed  as  test  results.  The  approach  can  be  judged  as 
achieving  validity  if  realistic  solutions  are  provided.  If  not,  we  need 
to  determine  why  and  where  it  needs  to  be  modified.  In  this  way,  the 

P 

state-of-the-art  of  evaluating  C  systems  can  be  advanced.  This  test 
concept  will  be  discussed  after  each  chapter  is  summarized  below. 

Chapter  2  started  with  the  necessary  definitions.  First,  is: 

Command  and  Control:  As  defined  in  JCS  Pub  1  : 

"The  exercise  of  authority  and  direction  of  a  properly  des¬ 
ignated  commander  over  assigned  forces  in  the  accomplishment  of  his 
mission.  Command  and  control  functions  are  performed  through  an 
arrangement  of  personnel,  equipment,  communications,  facilities,  and 
procedures  vrfiich  are  employed  by  a  commander  in  planning,  directing, 
coordinating  and  controlling  forces  and  operations  in  the 
accomplishments  of  his  mission." 

2 

A  C  System  has  two  components  (a.  and  b.  below),  when  at  rest,  and  a 
dynamic  state  (c.  below): 

a.  Physical  Entitles:  Equipment  (computer  and  peripherals, 
modems,  antennas,  local -area  networks),  software,  facilities, 
and  people. 

b.  Structure :  The  arrangement  and  interrelationships  of  physical 
entities,  standard  operating  procedures,  protocols,  concepts  of 
operation,  and  information  patterns.  (Structure  frequently 
reflects  doctrine  and  may  be  scenario  dependent.)  Such 
arrangements  are  often  physical  and  temporal. 

2 

c.  C  Process:  Refers  to  the  system  in  its  dynamic  state.  It 

says:  "What  is  the  system  doing?"  It  reflects  functions 
carried  out  by  the  system — sensing,  assessing,  generating, 

and  selecting  alternatives,  i.e.,  the  behavior  of  the  system. 
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System  Boundaries;  The  "boundary  of  a  system"  is  defined  as  a 
function  of  the  analysis  at  hand,  and  is  the  delineation  between  the 
system  being  studied  and  its  environment. 

Definitions  of  measures  applied  to  command  and  control  are  pre¬ 
sented  below: 

2 

a.  Measured/Specified  Inside  the  Boundary  of  the  C  System; 

1  .  Dimensional  Parameters  -  The  properties  or  characteristics 
inherent  in  the  physical  entities  whose  values  determine 
system  behavior  and  the  structure  under  question,  even  when 
at  rest  (size,  weight,  aperture  size,  capacity,  number  of 
pixels,  luminosity). 

2.  MOP  -  These  are  also  closely  related  to  inherent 
parameters  (physical  and  structural)  but  measure 
attributes  of  system  behavior  (gain  throughput,  error 
rate,  signal-to-noise  ratio). 

b.  Measures/Specified  Outside  the  Boundary  of  the  System; 

1  .  MOE  -  Measure  of  how  the  system  performs  its 

functions  within  an  operational  environment  (proba¬ 
bility  of  detection,  reaction  time,  number  of  targets 
nominated,  susceptibility  of  deception). 

c.  Measures/Specified  Outside  the  Boundary  of  the  Force: 

1 .  MOFE  -  Measure  of  how  a  system  and  the  force 

(sensors,  weapons,  system)  of  which  it  is  a  part 
performs  missions. 

Chapter  3  emphasizes  the  use  of  measures  in  the  conceptual  as 
well  as  the  implementation  categories  of  system  development.  We 
have  stressed  the  conceptual  area  since  we  believe  it  has  not 
received  sufficient  attention  in  the  past.  The  application 
categories  identified  in  Chapter  3  are  shown  below: 


Conceptual 


Doctrine  Development 

Requ irement s  Gener at ion/Val idat ion 
2 

C  Contribution  to  Force  Effectiveness 
New  Technology  Assessment 
R&D  Goals 
Implementation 

POM/Budget  Process 
Acquisition  Process 
Technical  Evaluation 
Operational  Evaluation 

In  Chapter  4,  a  conceptual  model  of  the  C^  process,  as  shown  in 
Figure  8-2,  has  been  created  from  several  progenitors,  primarily  the 
Lawson  model.  It  has  been  expanded  in  terms  of  both  two-sided  and 
hierarchical  dimensions.  The  subprocesses  have  been  described  but 
not  quantified.  We  believe  that  this  generic  process  model  is  now 
reasonably  robust  and  can  serve  as  a  guide  for  the  development  of 
detailed  process  models  (where  necessary  for  a  particular  analysis) 
or  tests  that  will  generate  measures  far  more  relevant  than  the 
isolated  "system  at  rest"  measures  often  seen  in  the  past.  The 
steps  in  this  development  are  shown  in  Figure  8-3.  The  measures 
derived  should  have  the  characteristics  shown  in  Table  8-1  .  It  is 
important  to  note  that  the  process  model  does  not  model  system 
components  themselves.  This  may  be  necessary  in  some  analyses. 
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FIGURE  8-2 

CONCEPTUAL  C2  PROCESS  MODEL 


•  Understood,  Represent  C2  Process 
Model  •  Define  Boundary  of  C2  System 
Repuirements  •  Represent  Time  and  Internal  Dynamics; 
-  MOP,  MOE,  MOFE;  Human;  Information  Transfer 
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FIGURE  8-3 
APPROACH 


TABLE  8-1 


DESIRED  CHARACTERISTICS  FOR  MEASURES 


CHARACTERISTICS 

DEFINITION 

Mission  oriented 

Relates  to  force/system  mission. 

Discrimi natory 

Identifies  real  difference  between 
alternatives. 

Measurable 

Can  be  computed  or  estimated. 

Quantitative 

Can  be  assigned  numbers  or  ranked. 

Realistic 

2 

Relates  realistically  to  the  C 
system  and  associated  uncertainties. 

Objective 

Can  be  defined  or  derived,  independent 
of  subjective  opinion.  (It  is 
recognized  that  some  measures  cannot 
be  objectively  defined.) 

Appropriate 

Relates  to  acceptable  standards  and 
analysis  objectives. 

Sensitive 

Reflects  changes  in  system  variables. 

Inclusive 

Reflects  those  standards  required 
by  the  analysis  objectives. 

Independent 

Is  mutually  exclusive  with  respect  to 
other  measures. 

Simple 

Is  easily  understood  by  the  user. 
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Any  C  system  will  be  developed  through  a  life  cycle  which  is 
composed  of  three  phases:  (1)  design,  including  concept  definition 
and  development;  (2)  acquisition  and  development;  and  (3) 
operational.  The  objectives  of  the  three  phases,  respectively,  are 
to : 

a.  Develop  a  concept  design,  desirable  characteristics,  and 
broad  requirements  to  meet  mission  objectives  and/or 
specify  system  requirements,  and  perform  early  R&D. 

b.  Develop  detailed  designs  and  acquire  the  system. 

c.  Deploy  and  evaluate  the  operational  system  (and  sometimes 
improve  the  system  or  improve  how  the  system  is  used).  It 
is  important  to  realize  that  in  some  analyses  the  hardware 
and  software  aspects  of  a  system  under  analysis  can  be 
relatively  unimportant  compared  to  the  structure  chosen  and 
the  procedures  used  in  a  system. 

In  addition,  one  must  consider  the  level  of  analysis  being  per¬ 
formed  as  shown  in  Figure  8-4:  mission,  system,  or  subsystem. 
Mission  analyses  are  designed  to  address  the  contribution  of  the 
system  to  the  force  structure  of  which  it  is  a  part.  System 

p 

analyses  involve  assessment  of  the  C  system’s  ability  to  operate 
with  regard  to  some  standard.  Subsystem  analyses  are  limited  to 
some  component  of  the  system  and  measure  performance. 

Thus  to  properly  analyze  a  system,  the  objective  must  be 
determined  as  a  function  of  life  cycle  and  analysis  type.  This 
determination  will  also  yield  the  model  type  required  for  the 
analysis.  A  set  of  sample  measures  may  also  be  developed,  a  priori, 
from  which  to  select  the  desired  measure.  Specific  examples  of 
these  measures  are  presented  in  Chapter  6. 

Chapter  7  develops  both  a  probabilistic  model  and  a  determin¬ 
istic  model  for  generating  measures  in  a  structured  fashion.  The 
probabilistic  model  generates  a  stopping  rule  for  how  many  measures 
are  needed  to  fully  represent  a  system.  Although  more  conceptual 
than  computational,  this  rule  is  an  important  step  forward  in  the 
search  for  "how  much  is  enough?"  The  deterministic  model  shown  in 
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FIGURE  8-4 
TYPES  OF  ANALYSES 


Figure  8-5  separates  the  efforts  of  examining  the  tradeoffs 
2 

necessary  in  C  system  design  into  two  categories:  those  relevant 

to  the  engineer  and  those  important  to  the  operations  analyst.  This 

formula  for  cooperation  of  these  two  communities  may  also  represent 

2 

a  concept  that  will  make  future  C  measures  far  more  enlightening  to 

decisionmakers.  Thus,  the  Workshop  has  made  important  contributions 
2 

to  understanding  C  evaluation  from  a  mathematical  point  of  view. 

Summary  and  Proposal  for  the  Next  Step 

This  report  contains  an  important  proposal  for  an  evaluative 

2 

structure  and  many  insights  for  the  evaluation  of  C  systems. 

However,  the  proposed  structure  remains  to  be  tested.  Application 
studies  to  test  the  structure  are  needed.  These  studies  must  attack 
real  problems  using  existing  and  newly  developed  tools  guided  by  the 
framework  set  forth  by  the  evaluation  architecture.  If  the 
evaluation  structure  provides  realistic  solutions,  we  can  be  more 
confident  as  to  the  validation  of  the  approach.  If  it  is 
productive,  then  the  limits  of  its  productivity  should  be  tested. 
However,  if  it  is  not  useful,  then  we  need  to  determine  the  reasons 
and  how  it  should  be  modified.  Finally,  the  evolutionary  nature  of 
the  topic  precludes  considering  any  part  of  this  report  as  a 
finished  issue.  We  are  trying  to  develop  a  structure  that  will  be 
useful  to  the  community.  In  doing  so,  the  formulations  change  but 
maintain  a  common  thread. 

A  meeting  will  be  held  in  January  1986  to  test  the  architecture 
with  several  real  problems  confronting  Military  Service  and  joint 
arenas,  ranging  from  air  defense  to  strategic  nuclear  and  Navy 
battle  group  issues. 

In  order  to  provide  a  simplified,  testable  procedure  at  that 
meeting,  we  are  reconfiguring  the  architecture  of  this  report  into  a 
chain  of  modules  (Figure  8-6).  It  takes  the  application  objectives 
and  bounds  the  system  in  accordance  with  our  definitions.  It  then 
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Basic  Model  Components 
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FIGURE  8-5 

CONSTRUCTIVE  (MODULAR)  APPROACH 


APPLICATION 

OBJECTIVES 


FIGURE  8-6 

MODULAR  C2  EVALUATION  STRUCTURE  (MCES) 
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continues  with  the  conceptual  model  in  the  development  process 
described  in  the  Model  chapter  and  in  Figure  8-2  to  produce  a 
specific  process  model.  Next,  the  measures  of  performance, 
effectiveness,  and  force  effectiveness  are  selected  according  to  the 
criteria  presented  in  Chapter  6.  Data  generation  techniques  are 
then  applied:  models,  exercises,  and  subjective  evaluations.  These 
resulting  measures  are  tested,  combined,  and  summarized  as  discussed 
in  Chapter  7.  Finally,  these  results  are  returned  to  the  decision¬ 
maker  for  interpretation,  assessment,  and  possible  generation  of 
additional  alternatives.  This  more  workable  formulation  of  the 
material  in  the  report  and  in  our  next  Workshop  will  test  this 
approach. 

The  reader  should  be  reminded  that  the  endeavour  reported 
herein  is  a  dynamic  one.  Even  as  this  document  was  being  written 
the  editors  and  other  participants  were  evolving  the  concepts 
presented.  As  we  tried  to  even  out  this  report  to  reflect  the 
current  thinking  we  found  that  we  kept  rewriting.  We  have, 
therefore,  chosen  to  freeze  the  document  to  represent  November  1985, 
knowing  that  evolution  of  these  concepts  will  continue.  We  hope  to 
provide  the  community  with  a  testable,  although  transient,  formu¬ 
lation  for  the  evaluation  of  C^  systems.  While  theory  moves  on, 
this  will  present  a  structure  to  help  in  communicating  and  framing 
the  analyses  so  badly  needed  in  this  area. 
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MOE  WORKSHOP  SPECIAL  SESSION 
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Genesis  of  NPS  MOE  Workshop 
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Of  Performance  V  Upon  System  S 


Constructive  (Modular)  Approach 
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Assessment  of  Results 
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Possible  Next  Alternative 
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Requirements  for  Completion 
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Possible  Tasks  for  Future 
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APPENDIX  B 


GLOSSARY 


ADM  Advanced  Development  Model 

ATO  Air  Tasking  Order 


Command  and  Control 

Command,  Control,  and  Communications 

Command,  Control,  Communications,  and  Intelligence 


DCT  Digital  Communication  Terminal 

DOD  Department  of  Defense 


IC  Intergroup  Coordinator 

JCS  Joint  Chiefs  of  Staff 


MCES  Modular  Command  and  Control  Evaluation  Structure 

MOE  Measure  of  Effectiveness 

MOFE  Measure  of  Force  Effectiveness 

MOP  Measure  (Variable)  of  Performance 

mors  Military  Operations  Research  Society 

NPS  Naval  Postgraduate  School 

POM  Program  Objectives  Memorandum 

R&D  Research  and  Development 

SLBM  Sea- Launched  Ballistic  Missile 

TWA  Tactical  Warning/Attack  Assessment 

U.S.  United  States 

VFME  Variable  for  Measuring  Effectiveness 

WWDSA  Worldwide  Digital  System  Architecture 
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